burt_mcalpine

Pool from multiprocessing issues

Discussion created by burt_mcalpine on Jul 26, 2011
Latest reply on Jan 23, 2012 by majgis
I am trying to use pool from the multiprocessing to speed up some operations def worker(d) that happen once for each leayer in the mxd.  this on is hard coded to D:\TEMP\Untitled4.mxd.   it runs but only one at a time.  I can see it start the pool, but only on is being used.  any help would be great.  I am running it in arctool box in ArcMap and have unchecked run as process. like I said it runs, but only one at a time....

import arcpy
import os
import multiprocessing

def worker(d):
    # buffer layer by the below values
    bfs = [101, 200, 201, 400, 401, 750, 751, 1001,
           1500, 1501, 2000, 2001, 2500]
    for bf in bfs:
        Output = os.path.basename(d)[:-4] + "_Buffer" + str(bf) + ".shp"
        print "Buffering " + d + " at " + str(bf) + " Feet"
        if arcpy.Exists(d):
            arcpy.Buffer_analysis(d, "D:\\Temp\\" + Output, str(bf) + " Feet")
            arcpy.Project_management("D:\\Temp\\" + Output, "D:\\Temp\\Test\\" + Output, "C:\Program Files (x86)\ArcGIS\Desktop10.0\Coordinate Systems\Geographic Coordinate Systems\North America\NAD 1983.prj")
            arcpy.Delete_management("D:\\Temp\\" + Output)
        else:
            print "No Data"


if __name__ == '__main__':
   
   #Sets MXD
    mxd = arcpy.mapping.MapDocument("D:\TEMP\Untitled4.mxd")
    #mxd = arcpy.mapping.MapDocument("CURRENT")

    #set some environments needed to get the correct outputs
    arcpy.env.overwriteOutput = True
    arcpy.env.workspace  = "D:\TEMP\Test"
    arcpy.env.outputCoordinateSystem = "C:\Program Files (x86)\ArcGIS\Desktop10.0\Coordinate Systems\Projected Coordinate Systems\UTM\NAD 1983\NAD 1983 UTM Zone 16N.prj"

    # of processors to use set for max minus 1
    prc = int(os.environ["NUMBER_OF_PROCESSORS"]) - 1

    # Create and start a pool of worker processes
    pool = multiprocessing.Pool(prc)

    # Gets all layer in the Current MXD
    lyrs = arcpy.mapping.ListLayers(mxd)

    #Loops through every layer and gets source data name and path
    for lyr in lyrs:
        d = lyr.dataSource
        print "Passing " + d + " to processing pool"
        arcpy.AddMessage("Passing " + d + " to processing pool")
        pool.apply_async(worker(d))

Outcomes