Using multiprocessing

Question

Hello all

I am trying to run multiprocessing python from Pro toolbox.

The simple code below works in IDLE with no problem (in PyScripter too).

It uses the print() and not AddMessage and the output comes with main2 since the child processes does not see the main window.

Trying to  run it in Pro toolbox gives unclear error message.

Trying to run it from notebook open many Pro's  (do not try this at home with a number bigger then 6 !!)

There are a few posts and blogs that say it should be possible https://www.esri.com/arcgis-blog/products/arcgis-desktop/analytics/python-multiprocessing-approaches-and-considerations/?rmedium=blogs_esri_com&rsource=/esri/arcgis/2011/08/29/multiprocessing/ but nothin new.

anybody is using this?

Each function (my_body) is complitly seperated from the others.

Thanks

import multiprocessing
import arcpy

def my_body (parm):
    arcpy.AddMessage("func " + str(parm))
    print("func2 " + str(parm))
    return(parm)

if __name__ == '__main__':
    listNumbers = [(1,1),(2,2),(3,3),(4,4)]

p = multiprocessing.Pool(6)
    res = p.map(my_body, listNumbers)
    arcpy.AddMessage("main " + str(res))
    print("main2 " + str(res))
    p.close()
    p.join()

LewisTrotter · Answer

I am trying to do something almost identical to this and I'm getting a "cannot pickle my_body function" error. This is the same for an atbx and non-atbx toolbox script, although the latter also opens ~10 instances are arcgis pro (I assume due to the workers calling Pro.exe, not Python.exe).

Does anyone have a working solution for using multiprocessing in a Python toolbox? I need to use it for processing numpy arrays, not geoprocessors.

ModyBuchbinder · Answer

Hi Jeff

Thanks but you are missing the point.

In my real application I need each process to do something. At the most it should return True/false if it fail or not. I just use the AddMessage as arcpy command that will show that the process is running for my testing.

The main problem is that I cannot run this as a script from Toolbox in Pro.

Thanks

Anonymous User · Answer

Process.map returns an iterable of all individual results, so trying to print the whole res object could be getting wonky.  Maybe try:

for res in p.map(my_body, listNumbers):
    arcpy.AddMessage("main " + str(res))
    print("main2 " + str(res))

or this method of multiprocessing:

with mp.Pool(processes=3) as pool:
    jobs = []
    for tple in listNumbers:
        jobs.append(pool.apply_async(my_body, (tple,) ))

res = [j.get() for j in jobs]
    for r in res:
        arcpy.AddMessage('main ' + str(r))
        print('main ' + str(r))

You could also return a dictionary of errors to help troubleshoot.

def my_body (parm):
    rdict = {'parm': parm, 'Error': None}
    try:
        arcpy.AddMessage("func " + str(parm))
        print("func2 " + str(parm))

except Exception as ex:
        rdict['Error'] = ex
    return rdict

if __name__ == '__main__':
    listNumbers = [(1,1),(2,2),(3,3),(4,4)]

p = multiprocessing.Pool(6)
    for res in p.map(my_body, listNumbers):
        arcpy.AddMessage("main " + str(res['parm']) + ' Error: ' + res['Error'])
        print("main2 " + str(res['parm']) + ' Error: ' + res['Error'])
    p.close()
    p.join()