bluecarbon1

Multiprocessing (parallelism) in geoprocessing tasks

Discussion created by bluecarbon1 on Jul 19, 2012
Latest reply on Jul 20, 2012 by bharold-esristaff
I am attempting to run a Python-based script which splits Landsat scenes in to vegetation, non-vegetation, cloud, water/shadow and snow/ice, based on equations in Oreopoulos et al. (2011).  The issue I�??m encountering has a documented solution in the script itself, which makes reference to the following archived forum post: http://forums.esri.com/Thread.asp?c=93&f=1729&t=284041
This forum led to a revision of the script; however, the lingering issue is associated with using "time" as described in the following post from the same forum:

Subject     Re: Multiprocessing (parallelism) in geoprocessing tasks
Author     Peter Smith
Date     Jun 15, 2009
Message     Thanks everybody for helping me out with this problem. Chris, your suggestion worked like a charm!
I'll add my addition in case others stumble across this: I had to switch from using time to something else, as the processes are all started at once. But after I created TEMP directories for every job,
it worked just fine.
Peter Smith
University of Arkansas

It seems Mr Smith was able to get his script running once he switched from using time to something else, but I am not sure what to use instead.  I have contacted the author and await their reply, but below is the text:

#---------------------------------------------------------------------------
#create temporary directories and a scratch geodatabase
#---------------------------------------------------------------------------
'''
Set the TEMP and TMP environment variables to different directories each time it is run
thereby preventing parallel processes from writing to the same TMP/TEMP workspace.
source: http://forums.esri.com/Thread.asp?c=93&f=1729&t=284041
'''
tempDir = 'C:/temp/gptmpenvr_' + time.strftime('%Y%m%d%H%M%S')
os.mkdir(tempDir)
os.environ["TEMP"] = tempDir
os.environ["TMP"] = tempDir

#create a scratch geodatabase
arcpy.CreateFileGDB_management(tempDir, "scratch.gdb", "10.0")
scratchFgdb = tempDir + "/scratch.gdb"


#---------------------------------------------------------------------------


Many thanks for your time

Outcomes