If I want to run 15 spatial joins at the same time, would it be better to use threading or multiprocessing? I've implemented threading already in a test and it cut 50 minutes (running sequentially) down to 15 minutes (running in concurrent threads). What is best practice in this case? Threading seems to work just dandy but if its not the recommended module for concurrent processing with ArcGIS I'll make the minor changes needed.
If threading works for you, stick with threading. I'd recommend multiprocessing over threading, though, for the following reasons:
ArcObjects instances (and therefore most arcpy/arcgisscripting objects) cannot cross thread boundaries. That is, if you create an object in one thread, you cannot access it in another.
On 32 bit operating systems, each individual process gets its own memory space, allowing each to consume a full 2(-4) gigs without having to share with other tool calls.
Robustness. If one process crashes, the others continue to run and you can restart the one that failed.
If threading works for you, stick with threading. I'd recommend multiprocessing over threading, though, for the following reasons:
ArcObjects instances (and therefore most arcpy/arcgisscripting objects) cannot cross thread boundaries. That is, if you create an object in one thread, you cannot access it in another.
On 32 bit operating systems, each individual process gets its own memory space, allowing each to consume a full 2(-4) gigs without having to share with other tool calls.
Robustness. If one process crashes, the others continue to run and you can restart the one that failed.
