Community

How can I debug a script tool that is using multiprocessing?

657
3
05-20-2019 02:49 PM
KylePerri
by Esri Contributor
Esri Contributor
‎05-20-2019 02:49 PM

I have a script tool that does a lot of processing on large amounts of data and needs to split the work between multiple CPUs to be efficient; however, I am having trouble understanding how I can debug the work that is happening within each CPU. It seems that arcpy.AddMessage does not work inside multiprocessing and I can't seem to attach my debugger to the command window that is opened when I run the tool. When I try attaching none of my breakpoints are hit and it says no symbols have been loaded.

0 Kudos
3 Replies
JoshuaBixby
by MVP Esteemed Contributor
MVP Esteemed Contributor
‎05-20-2019 02:56 PM

You mention splitting "the work between multiple threads," but then you also state that Add Message "does not work inside multiprocessing."  Multi-threading and multi-processing are not one of the same.  Are you using Python multiprocessing — Process-based parallelism — Python 3.7.3 documentation ?  If so, what is the overall structure of your multiprocessing?  And, what geoprocessing tools or ArcPy classes are you using?

0 Kudos
KylePerri
by Esri Contributor
Esri Contributor
‎05-20-2019 03:02 PM

Sorry, you're right. I edited my post. I am using multiprocessing not multithreading.

0 Kudos
ChrisTognela
by
New Contributor II
‎05-20-2019 04:20 PM

You might find it easier to do this outside of the ArcMap/ArcGIS Pro application environment using a standalone script, and log to a separate file. As it happens, I put this up last night, which may be helpful:

https://community.esri.com/people/ctognelaesriaustralia-com-au-esridist/blog/2019/05/20/threading-an...

The bugs you're running into, are they because you're doing multi-process work? Or are they potentially problems in the data you're processing, etc--what I'm trying to say is, if you weren't multi-processing, would you still run into these issues? If they're the latter, then perhaps sticking with a single thread--that you can debug--while you work on solving those problems against a subset of the data would be useful. Adding multi-processing can make them difficult to track down.

Raymond Hettinger has a couple of good talks on concurrency. Here's one:

https://www.youtube.com/watch?v=9zinZmE3Ogk

Good luck.

Chris

0 Kudos