Multiprocessing script errors on geoprocessing line of code - INFADI (Missing Dir)

MichaelMitchell1

I have written a script that uses pool.map to process multiple netCDF files and store information in a table. Each process runs a function to process one year. Each year has it's own individual file geodatabase, table within that geodatabase, and mxd. I also set the default workspace and scratch workspace to that geodatabase. For example when the function loads the year 1979 it accesses the 1979 geodatabase, 1979 table within that geodatabase, and 1979 mxd. 1980 would access the 1980 geodatabase, 1980 table within that geodatabase, and 1980 mxd.

If I run 1 process everything works fine. If I try to run 2 or more I get Fatal Error (INFADI) Missing Directory. It always happens at random times on a random process.  A try catch phrase isn't catching the error.  The process is completely shutting down.  All the other processes continue and the process that crashes starts another year.

It happens here:

              try:                    deleteme = Raster(arcpy.gp.ExtractByMask_sa(WSILayerRas, AOIshape))               except:                    continue  

Does anyone have any thoughts?
Thanks

ChrisSnyder

Try adding this code right before the function/child process is called:

import time time.sleep(1.1) #insurance newTempDir = r"C:\temp\gptmpenvr_" + time.strftime('%Y%m%d%H%M%S') os.mkdir(newTempDir) os.environ["TEMP"] = newTempDir os.environ["TMP"] = newTempDir

http://forums.esri.com/Thread.asp?c=93&f=1729&t=284041#881375

This is the trick I use to get multiple instances of arcpy (or gp) to run in parallel... Without this: Random crashes due to concurrent processes attempting to write to the same TEMP file(s) at the same time.

I use the subprocess module, so this might not (???) work with multiprocessing... Execute this code right before the child process is called, or in the child process, before you import arcpy.