topic Multiprocessing a nested loop in Python Questions

Multiprocessing a nested loop

SG_Johnson — Wed, 10 Nov 2021 16:52:07 GMT

Hi everyone,

Can anyone help with using the Multiprocessing.Pool() module to get my nested loop script running via multi-processing? I tried doing it myself but couldn't work out how without getting errors.

I am working with very big datasets which take hours to days to run at each stage of the loop so am in need of a way to make this significantly faster.

See script below:

#Import system modules
import arcpy
from arcpy import env

# define list values
list1 = [ 'x', 'y', 'z']
list2 = [ 'a', 'b', 'c']

for x in range(len(list1)):
for y in range(len(list2)):
try:

#Check out the Network Analyst extension license
arcpy.CheckOutExtension("Network")

#Set environment settings
env.overwriteOutput = True

#Set local variables
inNetworkDataset = "D:/RoadNetwork.gdb/RoadNetwork_ND"

#verify layer name
outNALayerName = list1[x]
chunk = list2[y]

#set variables
impedanceAttribute = "Distance"
inFacilities = "D:/facilities/" + chunk + ".shp"
polygonBarriers = "D:/barriers/" + outNALayerName + ".shp"
outLayerFile = "D:/outputs/" + outNALayerName + "_" + chunk + ".lyr"

#Make a barrier feature layer
barriersLayer = arcpy.management.MakeFeatureLayer(polygonBarriers,"PolygonBarriers").getOutput(0)

#Create a new service area layer.
outNALayer = arcpy.na.MakeServiceAreaLayer(inNetworkDataset, outNALayerName,
impedanceAttribute, "TRAVEL_FROM", "60", "DETAILED_POLYS", "NO_MERGE", "DISKS",
hierarchy = "NO_HIERARCHY", poly_trim_value = "100", restriction_attribute_name = ["OneWay"])
outNALayer = outNALayer.getOutput(0)

subLayerNames = arcpy.na.GetNAClassNames(outNALayer)

facilitiesLayerName = subLayerNames["Facilities"]
polygonbarriersLayerName = subLayerNames["PolygonBarriers"]

#Create field mappings for loading barriers
fieldMappings = arcpy.na.NAClassFieldMappings(outNALayer,polygonbarriersLayerName)
fieldMappings["BarrierType"].defaultValue = 0

#Load the facilities and barriers
arcpy.na.AddLocations(outNALayer, facilitiesLayerName, inFacilities, "", "")
arcpy.na.AddLocations(outNALayer, polygonbarriersLayerName, polygonBarriers, fieldMappings)

#Solve and save the service area layer
arcpy.na.Solve(outNALayer)
arcpy.management.SaveToLayerFile(outNALayer,outLayerFile,"ABSOLUTE")

except Exception as e:
# If an error occurred, print line number and error message
import traceback, sys
tb = sys.exc_info()[2]
now = datetime.now()
print now.strftime("%d%m%Y-%H%M%S")
print "An error occurred on line %i" % tb.tb_lineno
print str(e)

Re: Multiprocessing a nested loop

JoeBorgione — Wed, 10 Nov 2021 17:11:39 GMT

I've taken the liberty to format your code so it's readable. While I don't have any experience with the Multiprocessing.Pool() module, there are a couple things you should probably consider in your code.

There are several functions that are repeated every time the loop(s) are iterated, and I wonder if this is what is causing you some problems. Perhaps you could include the errors.

For example, you only need to checkout Network Analyst once; you are checking it out everytime you go through the loop. You should be getting a fatal error everytime you encounter Line 34; makeFeatureLayer because that feature layer is created on the first pass through the loop(s) and already exists in subsequent trips through loops. What is going on in lines 24 & 25: I don't understand your use of index [x] and [y] in the respective list.

My suggestion is to clean up your code such that you are not repeating yourself every time you go through it and see what happens.

#Import system modules import arcpy from arcpy import env # define list values list1 = [ 'x', 'y', 'z'] list2 = [ 'a', 'b', 'c'] for x in range(len(list1)): for y in range(len(list2)): try: #Check out the Network Analyst extension license arcpy.CheckOutExtension("Network") #Set environment settings env.overwriteOutput = True #Set local variables inNetworkDataset = "D:/RoadNetwork.gdb/RoadNetwork_ND" #verify layer name outNALayerName = list1[x] chunk = list2[y] #set variables impedanceAttribute = "Distance" inFacilities = "D:/facilities/" + chunk + ".shp" polygonBarriers = "D:/barriers/" + outNALayerName + ".shp" outLayerFile = "D:/outputs/" + outNALayerName + "_" + chunk + ".lyr" #Make a barrier feature layer barriersLayer = arcpy.management.MakeFeatureLayer(polygonBarriers,"PolygonBarriers").getOutput(0) #Create a new service area layer. outNALayer = arcpy.na.MakeServiceAreaLayer(inNetworkDataset, outNALayerName, impedanceAttribute, "TRAVEL_FROM", "60", "DETAILED_POLYS", "NO_MERGE", "DISKS", hierarchy = "NO_HIERARCHY", poly_trim_value = "100", restriction_attribute_name = ["OneWay"]) outNALayer = outNALayer.getOutput(0) subLayerNames = arcpy.na.GetNAClassNames(outNALayer) facilitiesLayerName = subLayerNames["Facilities"] polygonbarriersLayerName = subLayerNames["PolygonBarriers"] #Create field mappings for loading barriers fieldMappings = arcpy.na.NAClassFieldMappings(outNALayer,polygonbarriersLayerName) fieldMappings["BarrierType"].defaultValue = 0 #Load the facilities and barriers arcpy.na.AddLocations(outNALayer, facilitiesLayerName, inFacilities, "", "") arcpy.na.AddLocations(outNALayer, polygonbarriersLayerName, polygonBarriers, fieldMappings) #Solve and save the service area layer arcpy.na.Solve(outNALayer) arcpy.management.SaveToLayerFile(outNALayer,outLayerFile,"ABSOLUTE") except Exception as e: # If an error occurred, print line number and error message import traceback, sys tb = sys.exc_info()[2] now = datetime.now() print now.strftime("%d%m%Y-%H%M%S") print "An error occurred on line %i" % tb.tb_lineno print str(e)

Re: Multiprocessing a nested loop

DonMorrison1 — Wed, 10 Nov 2021 18:21:26 GMT

You should be able to get rid of your loops and let the python Multiprocessing Pool manage running the body of the loop. You have to prepare a list of "parameter sets" that the Pool will use to invoke the body whenever a processor becomes available. In your case the list would look something like this (no guarantee on the exact syntax)

parm_list = [('x','a'), ('x','b'),('x','c'),('y','a'),('y','b'),('y','c'),('z','a'),('z','b'),('z','c')]

then your body would be encapsulated in a function something list this

def my_body (parm): outNALayerName = parm[0] chunk = parm[1] .....

and you would call it like this

p = multiprocessing.Pool(<number of processors>) p.map(my_body, parm_list) p.close()

You have to be careful about lock conflicts, for instance if you use duplicate names for your temporary files or try have multiple processes updating the same file.