errors on performing "cell statistics" (error 99999) or "raster calculator" (000539) on 27 rasters

1602
19
02-03-2018 07:02 AM
yvanaltchenko
New Contributor

Hi, I need to do some statistics (mean, std) in order to compare each cell of 27 rasters. I get error using "cell statistics" (error 99999) or "raster calculator" (000539). Any ideas where come from the error (my formula in raster calculator has been rechecked and it is correct).

0 Kudos
19 Replies
XanderBakker
Esri Esteemed Contributor

You can write this code a lot shorter...

# Import arcpy module
import arcpy
import os

ws = r'D:\PhD\GIS\janvier_2018\GWIDC_calculation\Uncertainty_on_factor_A'
ras_name_template = 'GWDC_W1_A{0}.tif'
ras_list = [os.path.join(ws, ras_name_template.format("%02d" % (i,) )) for i in range(1, 28)]
ras_name_out = 'GWDC_W1_A_std.tif'
ras_path_out = os.path.join(ws, ras_name_out)

# Process: Cell Statistics
arcpy.gp.CellStatistics_sa(ras_list, ras_path_out, "STD", "DATA")
DanPatterson_Retired
MVP Esteemed Contributor

For Cell Statistics rasters are entered separated by commas

CellStatistics([InRas1, InRas2, InRas3], "SUM", "NODATA")

You might want to try that, or see the examples in the link on how to use it in a script to get you over the immediate need.

The raster calculator isn't meant to be used in scripts in any event

How Raster Calculator works so it might be best just to use the required format for scripting for Cell Statistics in order to get your Standard Deviation

NawajishNoman
Esri Contributor

I think the long string you created with the raster has some issue. It didn't have any separator between rasters. There could be other issues.

Please create a list of your rasters and see if it works. Here is an example.

import arcpy
from arcpy import env
from arcpy.sa import *
env.workspace = "C:/sapyexamples/data"
outCellStats = CellStatistics(["degs", "negs", "cost"], "STD", "DATA")
outCellStats.save("C:/sapyexamples/output/outcellstats.img")

OR

You can also specify the name of the raster in a variable and then construct a list as follows.

# Set variables
inRaster01 = "degs"
inRaster02 = "negs"
inRaster03 = "cost"

# Execute CellStatistics
outCellStatistics = CellStatistics([inRaster01, inRaster02, inRaster03], "RANGE", "NODATA")

Let us know if it doesn't work for you.

Thanks!

DanPatterson_Retired
MVP Esteemed Contributor

This is beginning to sound like another thread where the raster dimensions are physically fixed and there was a need for obtaining a statistical parameter over a time period.

https://community.esri.com/message/742193-re-how-to-calculate-the-percentile-for-each-cell-from-time...

In that case and in several others some success was had by emulating cell statistics within arrays.

  • What is the size of your tiff files .. ie rows and columns If they aren't too onerously bit, then could you zip them into a file for testing?

The procedure that has been used in several occasions for esri grids, tiffs, and ascii files is to

  • convert each raster to a numpy array ... the array will have 2 dimensions (cols, rows)
  • when stacked they produce a 3 dimensional array (depth, cols, rows)
  • the 3d array is what is used to calculate Cell Statistics, obtaining a parameter on a cell-by-cell basis over the time (depth) axis
  • This procedure has been used to calculate means, running means, percentiles etc.

So if the files are relatively small, they may be able to be loaded into arrays and processed in python/numpy escaping the limits imposed on memory by arcmap.

Perhaps on that note, I can't remember of you have tried PRO to do the same work.

So if you can share the data or even 1 or 2 files, I can try to set up a workflow that might work.

yvanaltchenko
New Contributor

Thanks for all your advice. Very much appreciate. My raster are quite big covering Africa with 0.005 degre cell size (about 800 Mo each for a total of 21 Go). Value of the 27 rasters are between 1 an 4.8 and are not time serie. They have been built from 8 parameters which are been firstly ranked into 5 value (1, 2 , 3 , 4 , 5) then weighted (equal weight of the parameter). The difference between the 27 rasters is that I applied 3 different rankings on 3 parameters. Then I need to calculcate mean and std of the 27 rasters in order to check areas where results are quite similar (it is a quite of sensitivity analysis).

I am not too familar with python script (but I can deal with modelbuilder) and very unfamilar with nunpy array (i basicly arcgis self-taught). I just wonder if my computer did not reach its capacity as the files are big (or ARCGIS capacity). I dont have PRO.

0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

You are probably running out of computing resources.  Run your system monitor to see how it progresses as you run a batch.

If memory is peaking, you will need to run in a 64 bit environment (ArcMap with background geoprocessing... although I don't know if it is used by SA off hand.  Or ArcGIS PRO)

Batching may be your only solution.

curtvprice
MVP Esteemed Contributor

> ArcMap with background geoprocessing... although I don't know if it is used by SA off hand. 

ArcMap background processing is fully supported by Spatial Analyst. You do need to do the separate install, it isn't part of the main ArcMap install.

DanPatterson_Retired
MVP Esteemed Contributor

As an alternative, particularly in a memory starved environment, see

Cell Statistics made easy... raster data over time

XanderBakker
Esri Esteemed Contributor

In addition to the great post by Dan Patterson‌, since you have rasters that are large you should implement blocks, to read parts of the raster and not the entire raster as explained in code sample 2 on this help page: RasterToNumPyArray—ArcPy Functions | ArcGIS Desktop 

0 Kudos
yvanaltchenko
New Contributor

Thank you for all your advices. I manage to calculate the standard deviation and the mean of my 27 rasters by cutting my raster into pieces. Cell statistics works when raster size is less than 80 Mo and when I process cell statistics one by one (if i create a model to go quicker it doesnt work). I guess it is link to my laptop capacity even if when processing the ram is not use fully.

0 Kudos