topic Re: How can I debug a script tool that is running multiple threads? in Python Questions

How can I debug a script tool that is using multiprocessing?

KylePerri — Mon, 20 May 2019 21:49:47 GMT

I have a script tool that does a lot of processing on large amounts of data and needs to split the work between multiple CPUs to be efficient; however, I am having trouble understanding how I can debug the work that is happening within each CPU. It seems that arcpy.AddMessage does not work inside multiprocessing and I can't seem to attach my debugger to the command window that is opened when I run the tool. When I try attaching none of my breakpoints are hit and it says no symbols have been loaded.

Re: How can I debug a script tool that is running multiple threads?

JoshuaBixby — Mon, 20 May 2019 21:56:09 GMT

You mention splitting "the work between multiple threads," but then you also state that Add Message "does not work inside multiprocessing." Multi-threading and multi-processing are not one of the same. Are you using Python multiprocessing — Process-based parallelism — Python 3.7.3 documentation ? If so, what is the overall structure of your multiprocessing? And, what geoprocessing tools or ArcPy classes are you using?

Re: How can I debug a script tool that is using multiprocessing?

KylePerri — Mon, 20 May 2019 22:02:08 GMT

Sorry, you're right. I edited my post. I am using multiprocessing not multithreading.

Re: How can I debug a script tool that is using multiprocessing?

ChrisTognela — Mon, 20 May 2019 23:20:56 GMT

You might find it easier to do this outside of the ArcMap/ArcGIS Pro application environment using a standalone script, and log to a separate file. As it happens, I put this up last night, which may be helpful:

https://community.esri.com/people/ctognelaesriaustralia-com-au-esridist/blog/2019/05/20/threading-and-subprocessing-for-speed-and-resilience

The bugs you're running into, are they because you're doing multi-process work? Or are they potentially problems in the data you're processing, etc--what I'm trying to say is, if you weren't multi-processing, would you still run into these issues? If they're the latter, then perhaps sticking with a single thread--that you can debug--while you work on solving those problems against a subset of the data would be useful. Adding multi-processing can make them difficult to track down.

Raymond Hettinger has a couple of good talks on concurrency. Here's one:

https://www.youtube.com/watch?v=9zinZmE3Ogk

Good luck.

Chris