CROSS VALIDATION COMPARISON

[EifLeffie]

Dear All,

I am modelling a phenomenon using Geostat Analyst particularly the Kriging technique. I generated several models and is using the cross validation comparison to determine the best model (out of more than 20 models that I have generated). What I am doing is comparing the prediction error parameters (e.g. root mean square, everage std. error, etc.) of one model against the other. However, as I have many models, the process is taking some time. I would compare them two by two (manually) and then compare the best output with the other models.

I wonder if there is any other way (technique or method - using all the models at the same time and not two by two as I am currently doing) to facilitate the comparison between models that will result with the best option.

Thank you so much for any help you could give.

Cheers,
Eif

[SteveLynch]

If you are using ArcGIS 10 you could write a short Python script

see http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//00300000000z000000.htm

Steve

[EifLeffie]

Thanks Steve for the reply. I have not used this utility yet as I am not that familiar with programming but I think it's worth a try.

Best regards,
Eif

[SteveLynch]

You can also, if multiple geostatistical layers are in ArcMap, right click on a geostatistical layer and choose Compare.

[EifLeffie]

Hi Steve,

Thanks a lot for the reply. What I did was to copy all the parameters in the prediction error of each model and encode in excel. The process is a bit tedious but I can be sure of which model I could choose (eg. model with lowest mean, Root Mean Sqr and Ave. Std. Error is a bit large but gap is small, etc.).

It would be easier if only one parameter (e.g. mean) is to be compared, however, in choosing the best model, we have to consider the RMS and ASE values, gaps, etc.).

This I guess cannot be done in ArcGIS as this needs manual analysis and would be based from the best jugdment of the user.

Thanks a lot and have a great weekends!

Cheers,
Eif

[MeganWolf]

I need to do a similar batch cross validation.  I am having trouble with writing a script that incorporates all of the statistics associated with cross validation.  I am able to print the Root Mean Square error but am not able to produce Mean Prediction, Average Standard Error, Standardized Prediction Error and the Root Mean Square Prediction Error.

Thanks in advance for the help!

[SteveLynch]

These are interpolation method dependant. IDW, for example on has error. Kriging has many more.

Steve