I learned how to do this a specific way, which is what I am describing. I'm sure someone can come up with a much better way then me, but here is what I have.
First, I have a function set up in a separate file called my "Worker". This is the function that will get passed to my pool of workers. This particular function is for creating a one page PDF based off of a map series of parcels. I used this function in lots of other scripts.
import os, sys
import arcpy
arcpy.env.overwriteOutput = True
#(Path to your MXD, Path you want the document to be extracted to, name of the page used in the map series)
def publicWorker(mxdPath, outFolder, name):
"""
This is the function that gets called and does the work of exporting the DDP for the UPI specified.
Note that this function does not try to write to arcpy.AddMessage() as nothing is ever displayed. If the cexport succeeds then it returns TRUE else FALSE.
"""
try:
mxdFile = arcpy.mp.ArcGISProject(mxdPath)
cleanName = name.replace("/", "_") #Removes the slash due to it being an escape character
l = mxdFile.listLayouts()[0]
if not l.mapSeries is None:
ms = l.mapSeries
if ms.enabled:
ms.currentPageNumber = ms.getPageNumberFromName(name)
file = os.path.join(outFolder, cleanName)
ms.exportToPDF(file, "CURRENT", resolution=300)
ms.refresh()
del mxdFile
return True
except arcpy.ExecuteError:
# Geoprocessor threw an error
arcpy.AddError(arcpy.GetMessages(2))
report.write("Execute Error:", arcpy.ExecuteError)
except Exception as e:
tb = sys.exc_info()[2]
report.write("Failed at Line %i \n" % tb.tb_lineno)
report.write("Error: {} \n".format(e))
return False
Then I have the main multiprocessing script that creates the pool of workers and assigns them my tasks.
Here is a portion of the code - Before line 8, my code creates a list of the variables called 'idList' from a feature class that has all of the parcels that were edited and need their PDFs updated. That list is used to create the jobs for the worker pool.
import arcpy
import multiprocessing
from PublicWorker import publicWorker
mxdPath =
outFolder =
def createmaps_handler():
jobs = []
for name in idList:
jobs.append((mxdPath,outFolder,name))
arcpy.AddMessage("Public job list has " + str(len(jobs)) + " elements.")
# Create and run multiprocessing pool.
multiprocessing.set_executable(*installpath*, 'pythonw.exe')) # make sure Python environment is used for running processes, even when this is run as a script tool
arcpy.AddMessage("Sending to pool")
cpuNum = multiprocessing.cpu_count() # determine number of cores to use
print("there are: " + str(cpuNum) + " cpu cores on this machine")
with multiprocessing.Pool(processes=cpuNum) as pool:
res = pool.starmap(publicWorker, jobs) # run jobs in job list; res is a list with return values of the worker function
pool.close()
pool.join()
# If an error has occurred report it # count how many times False appears in the list with the return values
failed = res.count(False) # count how many times False appears in the list with the return values
if failed > 0:
arcpy.AddError("{} workers failed!".format(failed))
1))
#makes sure code of the if-statement only executed for main process we start by running script, not the suprocesses created when using multiprocessing in this file
except arcpy.ExecuteError:
# Geoprocessor threw an error
arcpy.AddError(arcpy.GetMessages(2))
print("Execute Error:", arcpy.ExecuteError)
except Exception as e:
# Capture all other errors
arcpy.AddError(str(e))
print("Exception:", e)
if __name__ == '__main__':
createmaps_handler()
Eventually, I'd like to improve the error reporting in this, if possible, and I also have been working on a section for reporting how long it takes to run a batch of jobs. Hope this helps! A lot of this was adapted from lessons I learned during the class I have listed above.
