threading vs multiprocessing for concurrent gp tool runs

678
2
Jump to solution
11-25-2013 11:14 AM
ChrisMathers
Occasional Contributor III
If I want to run 15 spatial joins at the same time, would it be better to use threading or multiprocessing? I've implemented threading already in a test and it cut 50 minutes (running sequentially) down to 15 minutes (running in concurrent threads). What is best practice in this case? Threading seems to work just dandy but if its not the recommended module for concurrent processing with ArcGIS I'll make the minor changes needed.
Tags (2)
0 Kudos
1 Solution

Accepted Solutions
JasonScheirer
Occasional Contributor III
If threading works for you, stick with threading. I'd recommend multiprocessing over threading, though, for the following reasons:


  • ArcObjects instances (and therefore most arcpy/arcgisscripting objects) cannot cross thread boundaries. That is, if you create an object in one thread, you cannot access it in another.

  • On 32 bit operating systems, each individual process gets its own memory space, allowing each to consume a full 2(-4) gigs without having to share with other tool calls.

  • Robustness. If one process crashes, the others continue to run and you can restart the one that failed.


View solution in original post

0 Kudos
2 Replies
JasonScheirer
Occasional Contributor III
If threading works for you, stick with threading. I'd recommend multiprocessing over threading, though, for the following reasons:


  • ArcObjects instances (and therefore most arcpy/arcgisscripting objects) cannot cross thread boundaries. That is, if you create an object in one thread, you cannot access it in another.

  • On 32 bit operating systems, each individual process gets its own memory space, allowing each to consume a full 2(-4) gigs without having to share with other tool calls.

  • Robustness. If one process crashes, the others continue to run and you can restart the one that failed.


0 Kudos
ChrisMathers
Occasional Contributor III
Thanks Jason. Its not much to switch between the two so Ill change over to multiprocessing.
0 Kudos