curtvprice

Temporary rasters in arcpy

Blog Post created by curtvprice Champion on Mar 3, 2017

When you are working arcpy map algebra it is useful to have a more complete understanding of how temporary rasters are created and how they are managed. The help articles on arcpy map algebra and the Raster object are helpful but leave out some details that I believe are good to know if you are making heavy use of arcpy map algebra.

 

  1. The ArcGIS raster engine (whether you access it through the Raster Calculator tool, arcpy, or ArcObjects) will delay execution of local operators (examples: Plus, Multiply, Divide) until the output is needed for processing a non-local operator (for example to run Zonal Statistics) or to save a raster with the .save() method.  
  2. Temporary rasters are written to the current env.scratchWorkspace location. If this workspace is a geodatabase, temporary rasters are saved in FGDB format. If it is a folder, either Esri GRID (pre-10.5) or TIFF (10.5 or later) is used. If it is not set, the user's Default.gdb (C:\Users\username\ArcGIS\Default.gdb) is used.  I recommend setting env.scratchWorkspace to a folder workspace for best performance and stability out of the raster tools. This is especially true in versions before 10.5, as bugs in the raster tools for handling non-GRID file formats have been fixed over time.
  3. When you run the .save() method on a temporary raster object, and the format and location is the same as the temporary raster (GRID pre-10.5, TIFF 10.5+) the raster file will rename, not copy, the temporary raster to permanent. (Usually this is not important, but if you have a large (say, > 1GB) raster you will notice the difference.)

 

The following Python dialog (ArcGIS 10.5) should be instructive:

 

>>> import os
>>> import arcpy
>>> from arcpy import env
>>> from arcpy.sa import *
>>> arcpy.CheckOutExtension("Spatial")
u'CheckedOut'
>>> env.addOutputsToMap = False # I did this because I was using the ArcMap Pyt
on window
>>> testwk = r"C:\Users\cprice\Documents\ArcGIS\xxwk"  # empty folder
>>> env.workspace = testwk
>>> env.scratchWorkspace = testwk
>>> # set raster properties
... env.extent = "0 0 10 10"
>>> env.cellSize = 1
>>> def xdir():
...     from textwrap import fill as wrap
...     print(wrap(" ".join(os.listdir(env.workspace))))
...
>>> xdir() # scratchWorkspace is empty

>>> ras1 = CreateConstantRaster(0)
>>> xdir()  # temp raster written to disk
CreateConstantR1.tfw CreateConstantR1.tif CreateConstantR1.tif.aux.xml
CreateConstantR1.tif.vat.cpg CreateConstantR1.tif.vat.dbf
CreateConstantR1.tif.xml
>>> ras1 = CreateConstantRaster(0)
>>> xdir() # temp raster written to disk
CreateConstantR2.tfw CreateConstantR2.tif CreateConstantR2.tif.aux.xml
CreateConstantR2.tif.vat.cpg CreateConstantR2.tif.vat.dbf
CreateConstantR2.tif.xml
>>> ras2 = ras1 + 1
>>> ras3 = ras2 + 1
>>> xdir() # folder unchanged ("delayed execution")
CreateConstantR2.tfw CreateConstantR2.tif CreateConstantR2.tif.aux.xml
CreateConstantR2.tif.vat.cpg CreateConstantR2.tif.vat.dbf
CreateConstantR2.tif.vat.dbf.IGSKMCCWVM110W7.4604.4388.sr.lock
CreateConstantR2.tif.xml
>>> ras4 = FocalStatistics(ras3)
>>> xdir() # The tool above forced delayed execution of both Plus operations
CreateConstantR2.tfw CreateConstantR2.tif CreateConstantR2.tif.aux.xml
CreateConstantR2.tif.vat.cpg CreateConstantR2.tif.vat.dbf
CreateConstantR2.tif.vat.dbf.IGSKMCCWVM110W7.4604.4388.sr.lock
CreateConstantR2.tif.xml FocalSt_plus_ra1.tfw FocalSt_plus_ra1.tif
FocalSt_plus_ra1.tif.aux.xml FocalSt_plus_ra1.tif.xml plus_ras.tfw
plus_ras.tif plus_ras.tif.aux.xml plus_ras.tif.vat.cpg
plus_ras.tif.vat.dbf
plus_ras.tif.vat.dbf.IGSKMCCWVM110W7.4604.4388.sr.lock plus_ras1.tfw
plus_ras1.tif plus_ras1.tif.aux.xml plus_ras1.tif.vat.cpg
plus_ras1.tif.vat.dbf
plus_ras1.tif.vat.dbf.IGSKMCCWVM110W7.4604.4388.sr.lock
>>> ras4.save("ras4") # save ras4 in grid format
>>> xdir() # The tool above forced delayed execution of both Plus operations
CreateConstantR2.tfw CreateConstantR2.tif CreateConstantR2.tif.aux.xml
CreateConstantR2.tif.vat.cpg CreateConstantR2.tif.vat.dbf
CreateConstantR2.tif.vat.dbf.IGSKMCCWVM110W7.4604.4388.sr.lock
CreateConstantR2.tif.xml info plus_ras.tfw plus_ras.tif
plus_ras.tif.aux.xml plus_ras.tif.vat.cpg plus_ras.tif.vat.dbf
plus_ras.tif.vat.dbf.IGSKMCCWVM110W7.4604.4388.sr.lock plus_ras1.tfw
plus_ras1.tif plus_ras1.tif.aux.xml plus_ras1.tif.vat.cpg
plus_ras1.tif.vat.dbf
plus_ras1.tif.vat.dbf.IGSKMCCWVM110W7.4604.4388.sr.lock ras4
ras4.aux.xml
>>> for kk in [ras1, ras2, ras3, ras4]:
...     print kk.name, kk.isTemporary
...
None True
None True
None True
ras4 False
>>> # At end of python session all temporary rasters are deleted
>>> # only "ras4" and "info" will remain

 

Update, I have noticed at 10.5 that the old ArcGIS 9.3 MultiOutputMapAlgebra tool does not write TIFFs. This may be helpful if your tifs are so big that they pass the 2.1GB file size limit running arcpy x32 (say from ArcMap). MOMA is not supported and is tricky to use, as it only handles grids and the tool syntaxes are different (and some tools just aren't available), but it does still work at 10.5. This is not currently supported, use at your own risk!

cmd = "ras5 = focalmin(ras4)"
arcpy.gp.MultiOutputMapAlgebra_sa(cmd)
print(arcpy.GetMessages(0))
ras5 = Raster("ras5") # for later use of arcpy map algebra

Outcomes