topic Re: questions about cross-validation in ArcGIS GeoStatistical Analyst Questions

questions about cross-validation

pengjia — Wed, 05 Oct 2011 16:50:17 GMT

I have a question about the pic above.

Below is the original descriptions attached to this pic.

"For example, the root-mean-squared prediction error may be smaller for a particular model. Therefore, you might conclude that it is the optimal model. However, when comparing to another model, the root-mean-squared prediction error may be closer to the average estimated prediction standard error. This is a more valid model, because when you predict at a point without data, you have only the estimated standard errors to assess your uncertainty of that prediction. When the average estimated prediction standard errors are close to the root-mean-squared prediction errors from cross-validation, you can be confident that the prediction standard errors are appropriate. In the figure above, both kriging models are good, but those at the left are slightly better."

My question is, that, the ones with the root-mean-squared prediction errors closer to average standard errors should be the right one, instead of the left one. Why does it say that those at the left are slightly better? Thanks.

Re: questions about cross-validation

EricKrause — Thu, 06 Oct 2011 13:41:33 GMT

Good catch. I think this may be a typo in our help. I'll ask the author later today and get back to you.

If nothing else, we shouldn't be calling them both kriging models because the model on the right is from Local Polynomial Interpolation, which isn't a kriging model.

Thanks for the feedback.

Re: questions about cross-validation

EricKrause — Thu, 06 Oct 2011 16:35:56 GMT

Ok, we've talked it over, and we're going to change the graphic and some of the text from that topic. The 10.0 web help will update, and it will be changed in a future service pack as well as in version 10.1.

Thanks for bringing this to our attention.

Re: questions about cross-validation

pengjia — Fri, 07 Oct 2011 14:34:20 GMT

Ok, we've talked it over, and we're going to change the graphic and some of the text from that topic. The 10.0 web help will update, and it will be changed in a future service pack as well as in version 10.1.

Thanks for bringing this to our attention.

Thank you. So in addition to not being Kriging model on the right, according to the parameters in the cross-validatin, the one on the right should be better than the left one. Is that correct?

Re: questions about cross-validation

EricKrause — Fri, 07 Oct 2011 15:57:58 GMT

As for which is better, it's really a judgement call. Personally, I still like the model on the left because both the root-mean-square and average standard error are lower than the model on the right. A large difference between the RMS and the average standard error can indicate model problems, but a root-mean-square standardized of .85 indicates that the problem is not severe in this case. And the one point on the x-axis of the LPI model is also concerning.

When we change the graphic, we'll find an example where a lower RMS clearly does not imply a better model.

Re: questions about cross-validation

EricKrause — Thu, 06 Sep 2012 16:27:07 GMT

Update:

The topic has been updated with a better example of when a lower RMS does not indicate a better model:

http://resources.arcgis.com/en/help/main/10.1/index.html#//0031000000q0000000

Re: questions about cross-validation

PhilMorefield — Thu, 27 Sep 2012 20:31:14 GMT

As for which is better, it's really a judgement call. Personally, I still like the model on the left because both the root-mean-square and average standard error are lower than the model on the right. A large difference between the RMS and the average standard error can indicate model problems, but a root-mean-square standardized of .85 indicates that the problem is not severe in this case. And the one point on the x-axis of the LPI model is also concerning.

When we change the graphic, we'll find an example where a lower RMS clearly does not imply a better model.

Eric-

Is there a document that shows the various Prediction Error formulae (e.g., Mean Standardized Prediction Error, Root-Mean-Square Standardized, etc.), as implemented by ESRI?

I ask because different sources uses lightly different names for these metrics, and I know there can be subtle nuances to how they are calculated.

Re: questions about cross-validation

EricKrause — Thu, 27 Sep 2012 22:06:54 GMT

http://dusk.geo.orst.edu/gis/geostat_analyst.pdf

The formulas for the crossvalidation summary statistics can be found in Appendix A on page 273 (page 279 of the pdf, since the first six pages aren't numbered).

Re: questions about cross-validation

PhilMorefield — Fri, 28 Sep 2012 12:02:17 GMT

http://dusk.geo.orst.edu/gis/geostat_analyst.pdf

The formulas for the crossvalidation summary statistics can be found in Appendix A on page 273 (page 279 of the pdf, since the first six pages aren't numbered).

Perfect. Thanks Eric!

Re: questions about cross-validation

EricKrause — Fri, 28 Sep 2012 16:19:04 GMT

There's a typo in that pdf that I just noticed. For average standard error, the formula is missing a square. You can find the correct formulas here:

http://resources.arcgis.com/en/help/main/10.1/index.html#//00300000000z000000

"Average Standard Error" is the only formula that is different than you might expect. It might be better called "Root-Mean-Variance." We used this formula instead of a simple average because this formula is more directly comparable to the RMS.

Re: questions about cross-validation

PhilMorefield — Thu, 10 Jan 2013 19:35:14 GMT

There's a typo in that pdf that I just noticed. For average standard error, the formula is missing a square. You can find the correct formulas here:

http://resources.arcgis.com/en/help/main/10.1/index.html#//00300000000z000000

"Average Standard Error" is the only formula that is different than you might expect. It might be better called "Root-Mean-Variance." We used this formula instead of a simple average because this formula is more directly comparable to the RMS.

Okay, I can see how they are similar. But would there be any reason to compare the "Average Standard Error" of a EBK model to RMSE of an IDW model, for example? It seems like looking at RMSE for both is a more defensible comparison.

Also, are there some common/likely reasons for seeing a RMS Standardized Prediction Error that is very large?

Re: questions about cross-validation

EricKrause — Thu, 10 Jan 2013 20:58:30 GMT

Yes, you would want to compare the RMSE between EBK and IDW.

A large root-mean-square-standardized usually indicates the model is unstable. The most common reason for this is because the Gaussian semivariogram is very unstable if the nugget is very small, compared to the sill. Note that Stable with parameter=2 and K-Bessel with parameter=10 both correspond to the Gaussian semivariogram (it's a special case of both).

EDIT: Oh, I understand what you were asking. It doesn't make much sense to compare RMS and average standard error from different models, but it is useful to compare them within the same model because if the difference between them is large, it indicates that the model may have problems.

Re: questions about cross-validation

PhilMorefield — Mon, 14 Jan 2013 13:18:12 GMT

Yes, you would want to compare the RMSE between EBK and IDW.

A large root-mean-square-standardized usually indicates the model is unstable. The most common reason for this is because the Gaussian semivariogram is very unstable if the nugget is very small, compared to the sill. Note that Stable with parameter=2 and K-Bessel with parameter=10 both correspond to the Gaussian semivariogram (it's a special case of both).

EDIT: Oh, I understand what you were asking. It doesn't make much sense to compare RMS and average standard error from different models, but it is useful to compare them within the same model because if the difference between them is large, it indicates that the model may have problems.

Thanks Eric. I think I've got enough of an understanding to move forward with my project.

Re: questions about cross-validation

SirajMazumder — Thu, 20 Jun 2019 13:12:05 GMT

Well, folks,

I have a similar query. What does the minus sign in the mean error and mean standardized error indicate? I mean, how does one interpret such a result?

Thanks