topic Re: Using multiprocessing in Python Questions

Using multiprocessing

mody_buchbinder — Wed, 19 Apr 2023 11:11:50 GMT

Hello all

I am trying to run multiprocessing python from Pro toolbox.

The simple code below works in IDLE with no problem (in PyScripter too).

It uses the print() and not AddMessage and the output comes with main2 since the child processes does not see the main window.

Trying to run it in Pro toolbox gives unclear error message.

Trying to run it from notebook open many Pro's (do not try this at home with a number bigger then 6 !!)

There are a few posts and blogs that say it should be possible https://www.esri.com/arcgis-blog/products/arcgis-desktop/analytics/python-multiprocessing-approaches-and-considerations/?rmedium=blogs_esri_com&rsource=/esri/arcgis/2011/08/29/multiprocessing/ but nothin new.

anybody is using this?

Each function (my_body) is complitly seperated from the others.

Thanks

import multiprocessing import arcpy def my_body (parm): arcpy.AddMessage("func " + str(parm)) print("func2 " + str(parm)) return(parm) if __name__ == '__main__': listNumbers = [(1,1),(2,2),(3,3),(4,4)] p = multiprocessing.Pool(6) res = p.map(my_body, listNumbers) arcpy.AddMessage("main " + str(res)) print("main2 " + str(res)) p.close() p.join()

Re: Using multiprocessing

Anonymous User — Wed, 19 Apr 2023 12:58:54 GMT

Process.map returns an iterable of all individual results, so trying to print the whole res object could be getting wonky. Maybe try:

for res in p.map(my_body, listNumbers): arcpy.AddMessage("main " + str(res)) print("main2 " + str(res))

or this method of multiprocessing:

with mp.Pool(processes=3) as pool: jobs = [] for tple in listNumbers: jobs.append(pool.apply_async(my_body, (tple,) )) res = [j.get() for j in jobs] for r in res: arcpy.AddMessage('main ' + str(r)) print('main ' + str(r))

You could also return a dictionary of errors to help troubleshoot.

def my_body (parm): rdict = {'parm': parm, 'Error': None} try: arcpy.AddMessage("func " + str(parm)) print("func2 " + str(parm)) except Exception as ex: rdict['Error'] = ex return rdict if __name__ == '__main__': listNumbers = [(1,1),(2,2),(3,3),(4,4)] p = multiprocessing.Pool(6) for res in p.map(my_body, listNumbers): arcpy.AddMessage("main " + str(res['parm']) + ' Error: ' + res['Error']) print("main2 " + str(res['parm']) + ' Error: ' + res['Error']) p.close() p.join()

Re: Using multiprocessing

mody_buchbinder — Thu, 20 Apr 2023 04:18:14 GMT

Hi Jeff

Thanks but you are missing the point.

In my real application I need each process to do something. At the most it should return True/false if it fail or not. I just use the AddMessage as arcpy command that will show that the process is running for my testing.

The main problem is that I cannot run this as a script from Toolbox in Pro.

Thanks

Re: Using multiprocessing

LewisTrotter — Sat, 19 Aug 2023 08:18:39 GMT

I am trying to do something almost identical to this and I'm getting a "cannot pickle my_body function" error. This is the same for an atbx and non-atbx toolbox script, although the latter also opens ~10 instances are arcgis pro (I assume due to the workers calling Pro.exe, not Python.exe).

Does anyone have a working solution for using multiprocessing in a Python toolbox? I need to use it for processing numpy arrays, not geoprocessors.