topic Re: Need Help Automation of Kriging using Model builder or python - Reg. in ArcGIS GeoStatistical Analyst Questions

Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Thu, 12 Sep 2013 12:58:56 GMT

I am using arcmap 10.2.

My project deals with the prediction of PM and Ozone concentrations for 2004-2006 (1098 days) using known concentrations of them at
99 grids throughout the county.

I am trying to predict the unknown concentrations at mother's residences using ordinary kriging.

This is what I am doing:

1. Use Geostatistical Wizard
2. Select kriging cokriging
3. Input the Data layers and specifications
4. Select Finish
5. Read the model parameters and click OK
6. Generate Kriged predictions
7. Right click on Kriged data and convert to raster
8. Use spatial analyst tool Extraction
9. Select "Extract values to points"
10. Generate Extracted values of kriged predictions
11. Select conversion tool
12. Choose convert From Raster to Ascii
13. Generate the Ascii table.
14. Alternatively use Sample tool and input multiple rasters to generate an output table.

My question:

1. I need some guidance for automating this kriging process using model builder or python?

2. Is there any way to directly copy and paste the raster values to excel?

Can any one provide me a lead into this

Re: Need Help Automation of Kriging using Model builder or python - Reg.

SteveLynch — Thu, 12 Sep 2013 13:55:57 GMT

1) to automate the kriging process, have a look at http://blogs.esri.com/esri/arcgis/2010/06/18/automating-geostatistical-interpolation-using-template-layers/
2) for steps 10-14 do the following,
- you have a point feature class where the homes are, use this in the Extract values to points
- run the AddXY tool on the output feature class created above
- select all the rows in the output table and export, or if the output was sent to a shapefile you can use Excel to open the .dbf file

Hope this helps
Steve

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Thu, 12 Sep 2013 14:32:27 GMT

Hi Steve,

Thank you very much for your prompt response.

Thank you for guiding me with steps 10-14.

I will try them.

Also, I will try model builder or python scripting and get in touch with you if I have any difficulty.

Thank you

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Thu, 12 Sep 2013 14:43:19 GMT

Hi Steve,

Going back to Steps 10-14, I tried to use the extract values to points tool and used the kriged prediction raster as input raster.

Extraction was successful.

I am trying to add xy layer as suggested by you by right clicking the extracted points shp file.

I am not finding the option for it.

Also I am trying to use the sample tool to integrate all the extracted points shp file to generate a single file. A table is generated.

However, I am not able to convert it into ascii or dbf. When i try to export it, I am getting a 0KB file that I am not able to access.

Can you suggest suitably?

Thank you

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

EricKrause — Thu, 12 Sep 2013 15:30:07 GMT

Rather than converting the geostatistical layer to a raster and extracting the points, you should just use the "GA Layer to Points" geoprocessing tool. It will bypass the creation of the raster, and merging all of the outputs will be easy.

Also, you might want to consider using Empirical Bayesian Kriging (EBK) instead of Ordinary Kriging. The results are generally more accurate, and EBK is implemented as a geoprocessing tool, so it is much simpler to automate.

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Thu, 12 Sep 2013 15:39:12 GMT

Thank you Eric!!! I will try running EBK and will keep you updated.

Thanks

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Thu, 12 Sep 2013 17:59:15 GMT

Hi Eric/Steve,

I have another question:

I use a 99 gridded (12 km x 12 km grid) Ozone PM input file to predict concentration at each mother's residence.

The "Extract values to Points" output table generates raster values at each mother's location.

Do I multiply or divide this value by 144 (12 x 12) to adjust for the gridded area.

Please let me know on this.

Thanks

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

EricKrause — Thu, 12 Sep 2013 18:04:26 GMT

You don't need to do any adjustment. Ozone is a continuous phenomenon, so its value changes everywhere. When you extract the points from a raster, all you're doing is asking for the ozone value at the closest cell center. You don't need to do any corrections for the size of the grid cell.

Again, you can eliminate any imprecision from using the closest cell center by using GA Layer to Points. It will calculate the ozone value at the exact point rather than looking to the closest cell center.

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Thu, 12 Sep 2013 18:09:30 GMT

Thank you for the clarification Eric

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

SteveLynch — Thu, 12 Sep 2013 18:36:09 GMT

I am trying to add xy layer as suggested by you by right clicking the extracted points shp file.

I am not finding the option for it.

It's a geoprocessing tool, http://resources.arcgis.com/en/help/main/10.1/index.html#//001700000032000000

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Sun, 15 Sep 2013 23:03:15 GMT

Hi Steve/Eric,

I have created a model for my project using a model builder and incorporated the tools for Emprical bayesian kriging. I could run the model successfully and get the output that I expected.

Now, I need to loop the model to change the Z field of my merge1.CSV input point file (containing PM values for each day from Jan 1 2004 to Dec 31 2006). to run all the tools continuously to generate outputs for the entire dataset.

How I can loop through the merge1.CSV input point file in my model builder?

Thanks

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Mon, 16 Sep 2013 15:04:03 GMT

Hi Steve/Eric,

I attach my python script for Emprical Bayesian Kriging.

I need to inlcude a last step for looping through the entire merge1.csv data (Jan 1 2004 to Dec 31 2006).

How do I do this looping?

Pls. suggest

Thanks

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

EricKrause — Mon, 16 Sep 2013 21:37:53 GMT

This topic explains what you need to know:

http://resources.arcgis.com/en/help/main/10.2/index.html#/Examples_of_using_iterators_in_ModelBuilder/00400000001n000000/ESRI_SECTION1_33EDB1C8829F4CF4BBD5DABAA3ED34DD/

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Wed, 18 Sep 2013 16:28:21 GMT

Hi Eric/Steve,

I am trying to follow the steps below for iteration of EBK:

# Create a list variable to store the names of all the fields in the air quality data table
fieldList = arcpy.ListFields(File path to your table)

# Loop through each field in the air quality table
for zField in fieldList:

    # Set output raster name to match the field name with an added EBK suffix
    outRaster = zField + "_EBK"

    # Run the EBK interpolation for each field.
    arcpy.EmpiricalBayesianKriging_ga(...)

When I try to run my model in python, arcmap 10.2 crashes... I do not know the reason.

Have you experienced this problem?

Can you suggest me suitably.

Re: Need Help in deciding between ordinary and Empirical Bayesian Kriging

AnushuyaRamakrishnan — Tue, 24 Sep 2013 14:22:52 GMT

I have written python scripts for Empirical bayesian kriging and I am using it to generate predicted values for my sample mothers.
However, there is a concern that EBK is not well established in the literature.

I need certain clarifications before I can proceed to apply EBK for my study:

1. What is the underlying model to estimate parameters in semivariogram in EBK?

2. What is the bayesian Prior /bayesian rule that is applied in EBK?

3. What is the basis of calling it more accurate?

Any clarifications for these questions would be highly appreciated.

Thank you

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

EricKrause — Tue, 24 Sep 2013 16:57:26 GMT

1. If no transformation is applied, the underlying model is an intrinsic random function, specifically a Power semivariogram model. If a transformation is applied, it is a simple kriging model with an exponential semivariogram.

2. The prior distribution is estimated from the data using restricted maximum likelihood (REML). This is the difference between "Bayesian" and "empirical Bayesian." In empirical Bayesian models, the prior is estimated from the data, and the posterior distribution is generated through simulations.

3. There are several reasons to think EBK will, in general, be more accurate. First, REML is known to be a better estimator of semivariogram parameters than weighted least-squares (which is what is used in other kriging methods). Second, EBK works on subsets, so it eases the stationarity assumption of kriging. Other kriging methods assume global stationarity. EBK, however, only assumes stationarity within subsets, so it can work effectively even in the presence of global nonstationarity, as long as the subsets are close to being stationary. Third, EBK does a better job of estimating kriging variances because it can account for error in estimating the underlying semivariogram (it does this with simulations); other kriging models assume the semivariogram is modeled absolutely correctly, which often results in standard errors that are too small.

Here is a reference for EBK (a longer paper with all the details is in the works):

Krivoruchko K. and Gribov A. (2014) Pragmatic Bayesian kriging for non-stationary and moderately non-Gaussian data. Submitted. In Mathematics of Planet Earth. Proceedings of the 15th Annual Conference of the International Association for Mathematical Geosciences. Eds: Pardo-Igúzquiza, E.; Guardiola-Albert, C.; Heredia, J.; Moreno-Merino, L.; Durán, J.J.; Vargas-Guzmán, J.A. Springer 2014, pp. 61-64.

(We're not sure why the date is 2014 because it's already been published)

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Tue, 24 Sep 2013 17:23:49 GMT

Hi Eric,

Thank you very much for the clarification.

This would help me a lot.

Thanks

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Wed, 25 Sep 2013 13:48:40 GMT

How do we automate ordinary kriging? Just was curious to know about it.

We need to generate range, sill and nugget values for each day of the study. Is there any way for me to generate those as a first step in python script and give the inputs in the subsequent step for kriging.

I am trying to figure out how this can be done?

Any suggestions/help would be highly appreciated.

Thank you

Anushuya

Re: Need Help Automation of Kriging using Model builder or python - Reg.

EricKrause — Wed, 25 Sep 2013 17:22:12 GMT

Automation of Ordinary Kriging can be done with a template model and the Create Geostatistical Layer geoprocessing tool. This blog will take you through the basics, but you'll need to do some extra work:

http://blogs.esri.com/esri/arcgis/2010/06/18/automating-geostatistical-interpolation-using-template-layers/

However, you'll need to use a template XML file rather than a geostatistical layer. You can save this XML file at the end of the Geostatistical Wizard (after you hit Finish, click "Save..." on the Method Report screen). You'll need to change various "auto" flags in this XML file. You can change them with the "Set Model Parameter" gp tool, or you can just open the XML file in a text editor and manually change them (this way is probably easier).

Read the following topic (be sure to read the whole thing because I make a mistake in my first post):
http://forums.arcgis.com/threads/75743-Automatically-generate-anisotropic-semivariogram-parameters

The other poster's question was about anisotropic parameters and multiple semivariogram models, so the solution is more complicated that what you need. See if you can figure it out from those two resources, but I'm happy to help you if you run into any problems.

Re: Need Help Automation of Kriging using Model builder or python - Reg.

AnushuyaRamakrishnan — Wed, 25 Sep 2013 17:37:45 GMT

Thank you Eric... I will try this and let you know if I run into trouble.

Thanks

Anushuya