How can I debug a script tool that is using multiprocessing?

Question

I have a script tool that does a lot of processing on large amounts of data and needs to split the work between multiple CPUs to be efficient; however, I am having trouble understanding how I can debug the work that is happening within each CPU. It seems that arcpy.AddMessage does not work inside multiprocessing and I can't seem to attach my debugger to the command window that is opened when I run the tool. When I try attaching none of my breakpoints are hit and it says no symbols have been loaded.

ChrisTognela · Answer

You might find it easier to do this outside of the ArcMap/ArcGIS Pro application environment using a standalone script, and log to a separate file. As it happens, I put this up last night, which may be helpful:

https://community.esri.com/people/ctognelaesriaustralia-com-au-esridist/blog/2019/05/20/threading-and-subprocessing-for-speed-and-resilience

The bugs you're running into, are they because you're doing multi-process work? Or are they potentially problems in the data you're processing, etc--what I'm trying to say is, if you weren't multi-processing, would you still run into these issues? If they're the latter, then perhaps sticking with a single thread--that you can debug--while you work on solving those problems against a subset of the data would be useful. Adding multi-processing can make them difficult to track down.

Raymond Hettinger has a couple of good talks on concurrency. Here's one:

https://www.youtube.com/watch?v=9zinZmE3Ogk

Good luck.

Chris

KylePerri · Answer

Sorry, you're right. I edited my post. I am using multiprocessing not multithreading.

JoshuaBixby · Answer

You mention splitting "the work between multiple threads," but then you also state that Add Message "does not work inside multiprocessing."  Multi-threading and multi-processing are not one of the same.  Are you using Python multiprocessing — Process-based parallelism — Python 3.7.3 documentation ?  If so, what is the overall structure of your multiprocessing?  And, what geoprocessing tools or ArcPy classes are you using?