topic Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values in Python Questions

ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Wed, 09 Mar 2016 22:08:03 GMT

Hey everyone, I've been having some issues using the cell statistics tool in a python script recently. The issue is occurring in a stand alone script. I am trying to calculate standard deviation across 30 tif's representing annual cumulative rainfall for the globe.

While most of the values in the resulting raster (also a tif) are correct, there are large pockets of data that have extremely high standard deviation values (around 2000 - 3000 when they should be around 150).

When I run the same analysis in ArcGIS using the cell statistics geoprocessing tool the output is correct and I do not see these irregular values.

Does anyone have any idea why this would be happening? I assumed at first it was something wrong with my script but how could only some of the values be incorrect? There are also no no data values in these issue areas that could be causing the miscalculation although there is a no data value set in all of the input rasters.

I am providing statistics on the two raster in case it is helpful to anyone.

script output:
min = 0
max = 6697.9404296875
mean = 75.83343676729601
std. dev. = 333.72000491046

arcmap output:
min = 0
max = 1507.4833984375
mean = 35.804817248754
std. dev. = 84.35107381816201

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

DanPatterson_Retired — Wed, 09 Mar 2016 22:19:48 GMT

are there nodata cells and what are their values and what option did you use to account for them in the cell statistics options

Cell Statistics—Help | ArcGIS for Desktop even if you are coding, you will need to ensure that the cells align and are of the same size and the same nodata value in order to compute properly

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

XanderBakker — Thu, 10 Mar 2016 00:10:47 GMT

If you have manually executed the tool, you can copy the python snippet from the Results window. This will give you the code you can use to perform the exact same operation from Python (assuming the you don't have any strange setting in you geoprocessing environment). If you execute that python command, does that yield the same output and the manually executed tool?

If so, could you post your code that you used to generate the result with Python?

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 14:28:25 GMT

I manually executed the tool and then copied the code snipped from the results window. I ran the provided python code in both the python environment within ArcMap and as a standalone script in PyScripter. Both of these resulted in the same raster that was created when I manually executed the tool. All of the values in these rasters appeared to be correct. Do you want to see the code provided by the manual execution of the tool or the python code that I wrote that is giving me incorrect results?

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 14:44:40 GMT

Dan, I'm sorry. I think you replied to a previous post that I had deleted. I wanted to delete it to change my answer after looking at the data again.

There are nodata values in the input rasters. They are being reported as -3.40282346639e+038 in the Layer Properties window. I am using the "DATA" option in my script.

I do believe that the rasters align. I checked and they do all have the same extent.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 15:18:22 GMT

I figured out what the issue was...definitely a mistake on my end.

The input rasters were 32 bit but not floating point. I reran the analysis using a copy of the data that was floating point and everything worked fine. Sorry for the confusion.

Xander Bakker Dan Patterson

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

DanPatterson_Retired — Thu, 10 Mar 2016 15:38:40 GMT

I wondered what was going on... I hope you saw my answer about the raster type and no data... I though I was losing my mind... well... more than I have already

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 15:49:21 GMT

Fortunately your reply that got deleted was sent to my email so I was able to read it. That's why I checked the input data again to make sure it was floating point. I forgot that you could have 32 bit integer rasters.

The particular analysis I am performing is using 30 rasters representing annual cumulative rainfall for the globe. I was asked to calculate coefficient of variation (possibly a simplified version) by dividing the standard deviation of the dataset by the mean of the dataset ((std / mean) * 100).

Is there anyway that I could use integer values as input for such a calculation? It would be nice to save some space on disk by saving the input at a lower bit depth. The values only range from 0 mm - around 12000 mm.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

DanPatterson_Retired — Thu, 10 Mar 2016 16:05:53 GMT

Integers are fine... since they are within the realm of interval/ratio data. There is no need to have a decimal point to perform "method of moments" calculations, your data are simply scaled to mm instead of meters or cm and the spacing between increments on the number scale are equal and have real/physical meaning. The only thing you can't do the maths on is nominal or ordinal data (ie classes or ranks)

I use numpy arrays to consolidate space and speed up raster processing, You can save large arrays in binary format and reload them much like you can with other raster data. There is a recent post (linkless at the moment) dealing with raster processing for climate data whcih gives some basics. If you are familiar with python and numpy then there is a small group dealing with arrays, data and python NumPy Repository and other esoterica in data processing

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 16:33:40 GMT

Thanks for the link to the NumPy repository. I'll definitely take a look at that.

It seems like the integer input is what was causing the extremely high (2500-ish instead of 150-ish) standard deviation values though? Or am I misunderstanding the issue. Sorry for all the questions. I'm just trying to understand what happened so I don't run into a similar issue again.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

DanPatterson_Retired — Thu, 10 Mar 2016 16:39:38 GMT

yes means and std devs are in the units of the data... which isn't a problem, just state your problem and report your results that way you could scale everything cm or meter though, since the scaling is proportional

xm = 10 mm std dev = 20 mm is the same as 1 cm, 2 cm ... 1m 2m. But you will only believe me when you do the math in a spreadsheet ...

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 17:24:31 GMT

Yes, I understand that mean and standard deviation would be reported in the units of the data. In my case all input data is in mm so the mean and standard deviation would also be in mm.

However, I'm asking about the large discrepancy I was seeing between the standard deviation that was calculated using integer input and floating point input (both in mm units). When my analysis was run using integer values I would get a standard deviation value for a particular pixel of around 2000. When I ran the analysis again using floating point input the value changed for the same pixel to around 300 - the value of around 300 being the correct standard deviation value.

Why would there be such a large difference between these two values? I would expect to see a small variance across the two calculations but nothing close to the difference I was seeing.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

SteveLynch — Thu, 10 Mar 2016 17:47:10 GMT

A good idea, initially, might be to work with a subset of your data, say 3 rasters and in ArcMap zoom to a single pixel and use identify to get the value of all 4 (3 input and 1 output from CellStats) at that location.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 18:28:53 GMT

I have been doing some more testing and found a couple of things. It looks like the issue is cropping up when the input rasters are below 32 bit. I must have been mistaken before when I said the input was 32 bit integer rasters. When input is 16 bit I am seeing the very high std dev values. When I do the same calculation using 32 bit floats or ints the issue disappears.

I am attaching two screen shots of an Identify I did on a pixel that I see as having issues. I included all 30 input and unfortunately I could not fit it all in one screen shot.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 19:51:00 GMT

Is there a possibility that since the input values are so large that when the standard deviation is being calculated the total (for calculating mean) is hitting the limit of what a 16 bit integer can hold (65,535)? That would greatly decrease the mean and lead to increased deviation in the calculation. It could also explain why the calculation works when using 32 bit rasters as input and only gives me incorrect values in some areas of the output when using 16 bit rasters as input.

Just floating the idea.

I realize the mean in the screen shots is correct for the given numbers but that was calculated in a separate call to the cell statistics tool just to show what the mean should be.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

DanPatterson_Retired — Thu, 10 Mar 2016 20:08:25 GMT

time to try something other than a tiff as the inputs to this process ... esri grid or numpy array...

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Thu, 10 Mar 2016 20:25:16 GMT

I converted all of my 16bit integer tifs to grids with integer values. The analysis ran correctly and gave me results identical to using 32 bit tifs as input.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

XuguangWang — Tue, 15 Mar 2016 21:13:53 GMT

For the 16 bit integer tifs, what is their NoData value? Are all the input rasters in the same cell size? (I am suspecting that this issue is related to NoData handling during the resampling). You may find out the NoData and Cell Size info from the layer properties dialog, as shown by the following screen capture:

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

DanPatterson_Retired — Tue, 15 Mar 2016 21:53:58 GMT

Sum 6,000+ rasters - Python script

6,000+ rasters the follow-up

plus other show the use of numpy arrays to perform those calculations for multidimensional data quickly accounting for nodata values of all types. Not much use this time, but if you are working with more than a raster or two covering an area, consider using this alternative. Sample code is included in the last link and in a recent blog post on raster reclassification I did in the Py Links.

Re: ArcGIS 10.2.2 arcpy Cell Statistics giving Incorrect Values

StephenPalka1 — Mon, 21 Mar 2016 16:08:40 GMT

Sorry it took so long to respond. Was busy with work. Here is a screen shot of a layers properties window. All of the input rasters have the same properties.