topic Re: 10,000 inputs for Cell Statistics? in Python Questions

10,000 inputs for Cell Statistics?

NMWater — Tue, 25 Oct 2016 14:22:03 GMT

Hi all,

I was wondering if it is possible to have 10,000 inputs for cell statistics? I keep getting an error, saying that I can't save the output as a TIF. I get the following: ERROR 010240.

Any ideas or suggestions on how to tackle this?

Thanks a million,

import arcpy
import os

arcpy.CheckOutExtension("Spatial")
arcpy.env.overwriteOutput = True
arcpy.env.mask = r'E:\Reference_ET\Verification\GCS_PRISM_4km.shp'

print "Loaded Pre-processing extents..."

TmaxWs = r'M:\PRISM\Daily\tmax'
TminWs = r'M:\PRISM\Daily\tmin'
outws = r'E:\Reference_ET\KT'
TmaxList = []
TminList = []

print "Completing Lists..."

for dirpath, dirnames, filenames in arcpy.da.Walk(TmaxWs, topdown=True, datatype="RasterDataset"):
   for filename in filenames:
      TmaxList.append(os.path.join(dirpath, filename))
      listfile = os.path.join(dirpath, filename)

print "\t- Successful \n"

print "Calculating Average Max Temp from 1984-2015..."
TmaxRas = arcpy.sa.CellStatistics(TmaxList, "MEAN")

print "\t- Completed Cell Statistics... \n"

TmaxRas.save(outws + r'\Tmax30yr.tif')
print "\t- Successful \n"

Re: 10,000 inputs for Cell Statistics?

DanPatterson_Retired — Tue, 25 Oct 2016 14:30:04 GMT

throw a print statement before

TmaxRas.save(outws + r'\Tmax30yr.tif')

just to confirm that the path is correct.

Options to try

a different location
a different file type
save to a geodatabase

One of those seems to work for the dozens of Dr Google suggestions.

Re: 10,000 inputs for Cell Statistics?

NMWater — Tue, 25 Oct 2016 22:06:48 GMT

Hi Dan,

I tried the following three actions and still received the same error. I'm wondering if this is due to the number of inputs? It's daily data from 1984 to 2015, almost over 11,000 rasters. I also tried just limiting the process to two years, 730 rasters and still received the same results. It is completing but fails at saving. Any ideas or other suggestions? Can it possibly be some settings I'm missing in the script?

print "\t- Completed Cell Statistics... \n"

Thanks for all your help!

Re: 10,000 inputs for Cell Statistics?

DanPatterson_Retired — Wed, 26 Oct 2016 00:01:36 GMT

Well I tend not to use cell statistics, but I use NumPy... fewer problems and more functionality.

/blogs/dan_patterson/2016/08/14/reclassify-raster-data-simply

https://community.esri.com/thread/173806

The latter is a similarly related problem which you might be interested in. the discussion from which it is branched is quite long, has some other options and there are other samples.

Re: 10,000 inputs for Cell Statistics?

NMWater — Wed, 26 Oct 2016 20:53:37 GMT

Thanks for your reply Dan!

I added a couple of more print messages and it seems to be something with the .save() function. It keeps failing at that line. Would there be a possible way to get a more specified error message? Say for example, is this being caused by temp files, not enough space, etc.

Regarding, NumPy and your method, I read the thread. Would the "aust_temperature_demo.py" be something that I could modify to get a mean tif of temperature?

Thanks!

Re: 10,000 inputs for Cell Statistics?

DanPatterson_Retired — Wed, 26 Oct 2016 21:41:01 GMT

Go to the Results window... copy the info from their and your error message

Using the Results window—Help | ArcGIS for Desktop

And yes it could be an option, but I have several distractions on the go now so you would be on your own for a bit.

In the interim, you should be able to parse your works into parts, then assemble.

I would also still like to see your file names and those results window error messages and the actual error message... those I can deal with, even though they may seem cryptic

Re: 10,000 inputs for Cell Statistics?

NMWater — Fri, 28 Oct 2016 14:58:36 GMT

Hi Dan,

After running the processes on ArcMap python window. I received the following error:

I clipped my images to my study area and just processed two years worth of data, the processed work. I think this is most likely a RAM issue, since I have 1 TB hard drive on my computer.

Thanks a million!

Francisco

Re: 10,000 inputs for Cell Statistics?

DanPatterson_Retired — Fri, 28 Oct 2016 16:02:28 GMT

Take the advice... shut everything else down except what you need and don't forget 32 bit ArcMap only uses 4 Gb or ram regardless how much you have. Another reason to use Pro and python 3.x. Memory is cheap when you compare it to lost or slowdowns in time (try 32 Gb or ram and get the best fully memory loaded video card you can afford)

Re: 10,000 inputs for Cell Statistics?

NMWater — Fri, 28 Oct 2016 22:04:43 GMT

Dan thanks a million!

Switched to GIS pro and Python 3.4. Worked like a charm!

Francisco