topic Re: how to prevent memory use increase in task manager during 'bad' parallel processing? in Geoprocessing Questions

how to prevent memory use increase in task manager during 'bad' parallel processing?

ColineByczek — Wed, 09 Mar 2016 17:52:50 GMT

Hi,

I am currently trying to perform a model from model builder over 600,000 feature classes in arcgis 10.2 (perhaps 10.2.1 or 10.2.2)

In this model I use these tools (several times): viewshed analysis, raster calculator, extract by attributes, extract by mask, buffer, and erase.

To speed up the calculation, I'd like to use parallel processing. But because of urgent deadlines I cannot afford to learn about the correct way to perform it.

I've seen that not all tools can be used in the parallel processing environment and honestly, I don't have the time to check all the tools I use or find alternatives and of course learn to debug the final model with parallelisation in it.

Instead, I was told to split the work and run it in several arcgis windows opened at a same time. This works actually pretty well.

But after a time, if I check the Windows Task Manager, the memory use will increase a lot and at the same time the calculation speed will decrease.

Eventually the model crash and I have to start over where it ended.

My question is: does anyone no where precisely this comes from? Or more importantly, how to solve the problem.

I run the models from arctoolbox, not from model builder, so intermediate data should be erased at each run of the loop.

Thanks in advance for the help!

Coline

Re: how to prevent memory use increase in task manager during 'bad' parallel processing?

DanPatterson_Retired — Wed, 09 Mar 2016 18:59:57 GMT

Do you see a drop in memory use before a new loop begins to reflect the removal of intermediate or unused objects?

Perhaps Delete—Help | ArcGIS for Desktop to explicitly delete needs to be added in the process. And you want to make sure your environments are set to overwrite outputs Using geoprocessing options to control tool execution—Help | ArcGIS for Desktop remembering that layers added and then removed can still remain linked

Re: how to prevent memory use increase in task manager during 'bad' parallel processing?

JayantaPoddar — Thu, 10 Mar 2016 04:57:33 GMT

What is the hardware configuration?

600,000 feature classes going through multiple processes is a huge task and memory consuming.

Have you installed 64-bit background processing package? Background Geoprocessing (64-bit)—Help | ArcGIS for Desktop

Re: how to prevent memory use increase in task manager during 'bad' parallel processing?

ColineByczek — Thu, 10 Mar 2016 11:20:35 GMT

Hello,

Thanks you both for your help.

I did not delete intermediary results within the model but their names are not indexed through the loop. The geoprocessing environment is set to overwrite, and I can see the files been overwritten within Windows explorer and the .lock files pop in and out.

I do not use layer views or table views within the model, however I did not delete the feature class layer which 'Iterate Feature Selection' directly outputs. Could that be the origin of the problem? I'll try deleting that layer after each run.

There was no memory release in the task manager between interations, however between crashes there was.

I am under arcgis 10.2 so 64-bit background processing is not available. I'll consider upgrading to 10.3. But I do not run the models in background processing because I want to be able to follow the progress.

My computer is quite powerful: 14 CPUs and 48Go of memory. Makes up for my lack of efficiency... I know that the task is huge but it was unavoidable. I am calculating scenic beauty within viewshed from each open point within a 5,000km² zone, at a 200m resolution. I could have decreased the resolution but I'd rather learn how to solve computation problems. Each run is 7s long when starting the loop, which seemed fine.

In the end, I could perform all calculations within 24 hours of run through 10 parallel sessions, and only one crash. But I'd rather not make the same mistakes twice...

If you have other recommendations, thanks again.

Coline

Re: how to prevent memory use increase in task manager during 'bad' parallel processing?

DanPatterson_Retired — Thu, 10 Mar 2016 11:28:21 GMT

32 bit arc-stuff only uses 4 GB so examine your memory profile carefully, I suspect that is the limit you are hitting

Re: how to prevent memory use increase in task manager during 'bad' parallel processing?

JayantaPoddar — Thu, 10 Mar 2016 12:23:08 GMT

And 64-bit background processing is available for ver. 10.2 as well