I have a script tool that does a lot of processing on large amounts of data and needs to split the work between multiple CPUs to be efficient; however, I am having trouble understanding how I can debug the work that is happening within each CPU. It seems that arcpy.AddMessage does not work inside multiprocessing and I can't seem to attach my debugger to the command window that is opened when I run the tool. When I try attaching none of my breakpoints are hit and it says no symbols have been loaded.
You mention splitting "the work between multiple threads," but then you also state that Add Message "does not work inside multiprocessing." Multi-threading and multi-processing are not one of the same. Are you using Python multiprocessing — Process-based parallelism — Python 3.7.3 documentation ? If so, what is the overall structure of your multiprocessing? And, what geoprocessing tools or ArcPy classes are you using?
Sorry, you're right. I edited my post. I am using multiprocessing not multithreading.
You might find it easier to do this outside of the ArcMap/ArcGIS Pro application environment using a standalone script, and log to a separate file. As it happens, I put this up last night, which may be helpful:
The bugs you're running into, are they because you're doing multi-process work? Or are they potentially problems in the data you're processing, etc--what I'm trying to say is, if you weren't multi-processing, would you still run into these issues? If they're the latter, then perhaps sticking with a single thread--that you can debug--while you work on solving those problems against a subset of the data would be useful. Adding multi-processing can make them difficult to track down.
Raymond Hettinger has a couple of good talks on concurrency. Here's one:
https://www.youtube.com/watch?v=9zinZmE3Ogk
Good luck.
Chris