topic Re: Python fails to complete task (no error) in Python Questions

Python fails to complete task (no error)

BenLeslie1 — Tue, 17 Nov 2015 08:46:06 GMT

I'm attempting to Dissolve a feature class, which contains almost 2 million features, using a stand alone script. However the script gives up before the dissolve is complete and the output feature class covers approx 75% of the input features - there is no error.

When I perform the same dissolve in ArcMap, using either the tool GUI or the python window, it completes fully. So what gives? Is there something I can do to stop the stand alone script finishing early? - it is intended to become a longer script but at the moment it's set up to just perform the dissolve using all default settings.

(Arc 10.2)

Re: Python fails to complete task (no error)

DanPatterson_Retired — Tue, 17 Nov 2015 10:55:57 GMT

Perhaps you are running out of memory, which the ArcMap memory caches to disk but the python script isn't able to.

How long does it take before it totally bails out on you? after a substantial period of time or fairly soon?
Are you watching memory consumption while the script is running?
Did you completely delete previous attempts?
Destinations of file to a local disk or a network?
other pertinent information?

Re: Python fails to complete task (no error)

BenLeslie1 — Wed, 18 Nov 2015 10:45:40 GMT

OK I tried this dissolve a few more times (I did not delete previous attempts but am outputting to a different file each time) -

if I run only the script:

successful after approx 9 mins
Used physical memory around 21% throughout (max approx 25%)
Running on local disk

if I also have ArcMap open:

bailed out after approx 9 minutes - coverage of output is approx 75% of what it should be if I just have ArcMap open without using it; however if I do things like loading imagery off the network at the same time as running the script the output can be more like 50% of what it should be - no errrors are reported.
Used physical memory around 21% throughout (max approx 25%)
Running on local disk

My computer is Intel Xeon E5-2620 2GHz (2 processors)

16 GB RAM

24 cores

This behaviour is rather disturbing to me - it undermines the use of python as a reliable method of geoprocessing - how do I catch these errors?

I thought that one of the advantages of using scripts was that I could do other things while they run in the background.

Re: Python fails to complete task (no error)

DanPatterson_Retired — Wed, 18 Nov 2015 11:15:34 GMT

It may depend on where you are running your scripts. Are you using the python ide that is embedded in ArcMap? That may produce different results that running an ide separately (such as Pyscripter or PythonWin etc). There are more questions on this background vs foreground processing... it almost rivals Define Projection vs Project.

In any event, this works for me.. I never work on, or use in a process, any file loaded in ArcMap. If it has a table, I close it if open. If it is a large file, I remove the layer,then I incrementally save the project... I do python stuff in a separate IDE not the one built into ArcMap or Pro. If I am using ArcMap this entails using python 2.7. Since I have Pro installed, I have python 3.4 installed in which case, I can switch over to Pro and/or python 3.4 and let Arcmap and python 2.7 chug along. I have never had issues, but I have only done dissolved on less than 1Gb, I prefer to tile anything larger either spatially or by attribute and reassemble the bits when a combined file is needed. There are links (which I can't find now) on how to work with large data sets, ArcMap will only use 4 Gb of memory regardless of what you have installed so tiling in some form is often the solution whether using tools or by explicitly splitting your datasets and reassembling. Sorry I can't be more helpful, but that's all I have.

Re: Python fails to complete task (no error)

DanPatterson_Retired — Wed, 18 Nov 2015 23:52:57 GMT

Arecent post, updating an old post suggest ArcGIS Pro, may solve this problem Solution to: Dissolve tool does not fully dissolve medium-size dataset

The Dissolve help topic... Dissolve—Data Management toolbox | ArcGIS for Desktop

Tiling large datasets... Tiled processing of large datasets—Appendices | ArcGIS for Desktop

Now you can Dice it yourself Dice—Data Management toolbox | ArcGIS for Desktop

And the recommendation...don't do anything else, when processing large data set.