understanding cross validation errors

Question

Hello ArcGIS community,

I'm trying to understand how different errors of cross validation are calculated. I've the formula of desktop.arcgis.com page, but I could not find out for some of them. I'm using the ozone tutorial data. I've attached 2 files. Thank you to answer to these questions :

1- How is calculated the standard error for an individual sample ? from wich data ? I've tried unsuccessfuly different combinaison

2- How is calculated the average standard error ? I've found a different value for the average of the individual standard errors.

But not for the stadardized error and the mean standardized, where I could recalculate standardized error from the standard error and the mean standardized from the mean of the individual standard errors.

3- How is calculated the RMS error ? is that from the whole prediction data or only from the cross validation data ?

4- which standard error we use for standardizing the RMS ?

Is it possible, if you have time, to make a summary of how calculate all these parameters, it can be usefull for the whole community, as understanding the prediction errors is as important than prediction itself.

Thank you so much for taking time to answer.

Capture1.JPG

Capture2.JPG

tables display2.jpg

CollectoriPad.jpg

FedericoRiet_Sapriza_1-1610398385141.png

JayantaPoddar_0-1646842111432.png

nacerazzouzi · Answer

It doesn't answer to my questions. But it's fair enough. Thank you anyway.

Is there any french version of the ArcGIS pdfs ?

Anybody can provide the exact french translation of :

- Root Mean Square Error

- Root Mean Square Standardized Error

- Average Standard Error

Regards

Eric6346 · Answer

Whenever kriging "predicts" at a new location, it calculates both a kriging prediction and a kriging variance (the standard error is the square root of the variance).  The formulas for these values are complicated, but they are based on the semivariogram and the locations/values of neighboring points.  You can see these formulas in the pdf, but they are not simple to explain and require understanding of linear algebra and Lagrange multipliers.

When a point is being cross-validated, its value is hidden, and the kriging prediction and kriging variance at the location of the hidden point is calculated based on every other point.  We call these the cross-validated prediction and cross-validated variance of the point.  The "Predicted" field of the table represents the cross-validated prediction, and the "Standard Error" field represents the square root of the cross-validated variance. The "Measured" field is the true value of the hidden point, and the "Error" field is the difference between the Predicted and Measured.

Once these statistics are calculated for every point by sequentially hiding each of them, the summary statistics like Average Standard Error can be calculated.  The Mean Error, for example, is the simple average of the Error column.  The Average Standard Error is the square root of the average of the cross-validated kriging variances.  To calculate it by hand, square all of the values of the Standard Error field, take the average of the squared values, and take the square root of the result.  You can see the formulas for all of these summary statistics in my first post.

In the formulas, z(s) refers to the measured value at a location.  z^(s) refers to the cross-validated prediction (the "hat" ^ appears above z(s)). σ-hat refers to the cross-validated standard error (the square root of the variance).

The actual computation of these numbers is less important than what they're measuring.  The RMSE and Average Standard Error are important because they both estimate uncertainty of predicted values in different ways, and if the uncertainty is being estimated consistently, both numbers should be approximately equal.  If they differ significantly, this is a sign that uncertainties are not being modeled effectively.  Similarly, the standardized RMS directly measures whether standard errors are being under- or over-estimated.

nacerazzouzi · Answer

Dear Erik,
I thank you deeply for your answers which I am sure are very clear. However, this is not exactly what could be of use to me. I do not intend to calculate the parameters by myself. But, until I did it for a little example, I couldn't figure it out.

So if I am still allowed to abuse your time, I pose my problem in another way. But I'm sure your answer will help a lot of people in the community.

From my experience in teaching, it is often the calculation of uncertainties that poses the most problems of understanding, and finally we find ourselves confronted during research work to see very nice models, but whose reliability and validity remains to be proven for lack of objective analysis of estimation errors.

then, if you can explain us through the example of the pdf that you sent (page 54) which explains very well the principle of calculation of the kriging until the final estimate for an unsampled point by finally giving its variance and its standard error (page 58). until then I understood perfectly, well I think.

Question 1- is it this standard error which is used to calculate the "average standard error" in "capture3.jpg"? is it the same in "Capture4.JPG"? if yes then it's ok.

Let's now take 1 point and give it an estimate by cross-validation: let's say sample (1,3) without doing the calculation let's say its new value is "104", with the error is 105-104=1.

Question 2: Could you calculate its standard error like in "Capture5.JPG"?

Question 3- How is calculated the RMS error "Capture3.JPG"? is that from the whole prediction data or only from the cross validation data?

Question 4- which standard error we use for standardizing the RMS "Capture3.JPG"?

sorry to bother you and thank you again ..

Cheers.