topic Re: Instability when running multiple models at once in Geoprocessing Questions

Instability when running multiple models at once

MichaelDavis3 — Sun, 22 May 2011 00:22:25 GMT

I'm running a model in ArcGIS 10 to iterate through ASCII LIDAR tiles and extract features based on a form factor algorithm. Due to the size of the dataset, I'm running 3 of the models concurrently in seperate instances of ArcCatalog.

When running the model on my local Windows XP machine the processes seem to crash after 120 or so iterations. On our Windows 2008 R2 server running CITRIX the processes seem to be more stable, but are still crashing after an extended period.

I've isolated the models as much as possible (each process reads and writes to its own file geodatabase, in_memory writes are to variables with unique names) but I'm still seeing crashes (not errors, but all out program failure).

Can anyone enlighten me on how ARC instances manage memory? Are they each using a seperate address space for in_memory writes?

Also, does anyone know if there are plans to implement multi-threading, or even distributed processing in future versions of ArcGIS/ ArcGIS Server? It would be really awesome if ESRI could link up ArcGIS clients with ArcGIS Server instances as host controlers for distributed computing.

Ideally something that could be implemented in Model builder so big jobs could be run aginst a server, which would kick pieces of the job out to idle machines running ArcGIS.

Thanks!

-Mike

Re: Instability when running multiple models at once

BruceHarold — Mon, 23 May 2011 13:53:39 GMT

Hi Mike

Your workflow is of interest; we have a few samples using Python's subprocess and multiprocessing modules to parallelize geoprocessing tasks, but nothing so far using Lidar. If you can share your data and processing requirements we may be able to figure out an approach for you. If you can do this, please email me at bharold@esri.com.

Regards

Re: Instability when running multiple models at once

LukeWebb — Tue, 24 May 2011 11:19:40 GMT

Are these samples on the web site somewhere?

Parallelised geoprocessing is where I want to go, I have some models that essentially meet near the end but are seperate processes until then. These should be easy to paralellise I hope?

Is it a feature of modelbuilder I havent found yet? (I can do some python also regardless)