nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

questions about cross-validation

pengjia

I have a question about the pic above.

Below is the original descriptions attached to this pic.

"For example, the root-mean-squared prediction error may be smaller for a particular model. Therefore, you might conclude that it is the optimal model. However, when comparing to another model, the root-mean-squared prediction error may be closer to the average estimated prediction standard error. This is a more valid model, because when you predict at a point without data, you have only the estimated standard errors to assess your uncertainty of that prediction. When the average estimated prediction standard errors are close to the root-mean-squared prediction errors from cross-validation, you can be confident that the prediction standard errors are appropriate. In the figure above, both kriging models are good, but those at the left are slightly better."

My question is, that, the ones with the root-mean-squared prediction errors closer to average standard errors should be the right one, instead of the left one. Why does it say that those at the left are slightly better? Thanks.

Vind meer berichten getagd met

Accepted answers

All comments

SirajMazumder

Well, folks,

I have a similar query. What does the minus sign in the mean error and mean standardized error indicate? I mean, how does one interpret such a result?

Thanks

PhilMorefield

Yes, you would want to compare the RMSE between EBK and IDW.

A large root-mean-square-standardized usually indicates the model is unstable. The most common reason for this is because the Gaussian semivariogram is very unstable if the nugget is very small, compared to the sill. Note that Stable with parameter=2 and K-Bessel with parameter=10 both correspond to the Gaussian semivariogram (it's a special case of both).

EDIT: Oh, I understand what you were asking. It doesn't make much sense to compare RMS and average standard error from different models, but it is useful to compare them within the same model because if the difference between them is large, it indicates that the model may have problems.

Thanks Eric. I think I've got enough of an understanding to move forward with my project.

EricKrause

Yes, you would want to compare the RMSE between EBK and IDW.

A large root-mean-square-standardized usually indicates the model is unstable. The most common reason for this is because the Gaussian semivariogram is very unstable if the nugget is very small, compared to the sill. Note that Stable with parameter=2 and K-Bessel with parameter=10 both correspond to the Gaussian semivariogram (it's a special case of both).

EDIT: Oh, I understand what you were asking. It doesn't make much sense to compare RMS and average standard error from different models, but it is useful to compare them within the same model because if the difference between them is large, it indicates that the model may have problems.

PhilMorefield

There's a typo in that pdf that I just noticed. For average standard error, the formula is missing a square. You can find the correct formulas here:

http://resources.arcgis.com/en/help/main/10.1/index.html#//00300000000z000000

"Average Standard Error" is the only formula that is different than you might expect. It might be better called "Root-Mean-Variance." We used this formula instead of a simple average because this formula is more directly comparable to the RMS.

Okay, I can see how they are similar. But would there be any reason to compare the "Average Standard Error" of a EBK model to RMSE of an IDW model, for example? It seems like looking at RMSE for both is a more defensible comparison.

Also, are there some common/likely reasons for seeing a RMS Standardized Prediction Error that is very large?

EricKrause

There's a typo in that pdf that I just noticed. For average standard error, the formula is missing a square. You can find the correct formulas here:

http://resources.arcgis.com/en/help/main/10.1/index.html#//00300000000z000000

"Average Standard Error" is the only formula that is different than you might expect. It might be better called "Root-Mean-Variance." We used this formula instead of a simple average because this formula is more directly comparable to the RMS.

PhilMorefield

http://dusk.geo.orst.edu/gis/geostat_analyst.pdf

The formulas for the crossvalidation summary statistics can be found in Appendix A on page 273 (page 279 of the pdf, since the first six pages aren't numbered).

Perfect. Thanks Eric!

EricKrause

http://dusk.geo.orst.edu/gis/geostat_analyst.pdf

The formulas for the crossvalidation summary statistics can be found in Appendix A on page 273 (page 279 of the pdf, since the first six pages aren't numbered).

PhilMorefield

As for which is better, it's really a judgement call. Personally, I still like the model on the left because both the root-mean-square and average standard error are lower than the model on the right. A large difference between the RMS and the average standard error can indicate model problems, but a root-mean-square standardized of .85 indicates that the problem is not severe in this case. And the one point on the x-axis of the LPI model is also concerning.

When we change the graphic, we'll find an example where a lower RMS clearly does not imply a better model.

Eric-

Is there a document that shows the various Prediction Error formulae (e.g., Mean Standardized Prediction Error, Root-Mean-Square Standardized, etc.), as implemented by ESRI?

I ask because different sources uses lightly different names for these metrics, and I know there can be subtle nuances to how they are calculated.

EricKrause

Update:

The topic has been updated with a better example of when a lower RMS does not indicate a better model:

http://resources.arcgis.com/en/help/main/10.1/index.html#//0031000000q0000000

EricKrause

As for which is better, it's really a judgement call. Personally, I still like the model on the left because both the root-mean-square and average standard error are lower than the model on the right. A large difference between the RMS and the average standard error can indicate model problems, but a root-mean-square standardized of .85 indicates that the problem is not severe in this case. And the one point on the x-axis of the LPI model is also concerning.

When we change the graphic, we'll find an example where a lower RMS clearly does not imply a better model.

pengjia

Ok, we've talked it over, and we're going to change the graphic and some of the text from that topic. The 10.0 web help will update, and it will be changed in a future service pack as well as in version 10.1.

Thanks for bringing this to our attention.

Thank you. So in addition to not being Kriging model on the right, according to the parameters in the cross-validatin, the one on the right should be better than the left one. Is that correct?

EricKrause

Ok, we've talked it over, and we're going to change the graphic and some of the text from that topic. The 10.0 web help will update, and it will be changed in a future service pack as well as in version 10.1.

Thanks for bringing this to our attention.

EricKrause

Good catch. I think this may be a typo in our help. I'll ask the author later today and get back to you.

If nothing else, we shouldn't be calling them both kriging models because the model on the right is from Local Polynomial Interpolation, which isn't a kriging model.

Thanks for the feedback.

Doe mee aan het gesprek

Aangemelde leden kunnen berichten plaatsen, updates volgen en meer. Nieuw hier? Registreer een gratis account.

Snelle Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.