I'm attempting to Dissolve a feature class, which contains almost 2 million features, using a stand alone script. However the script gives up before the dissolve is complete and the output feature class covers approx 75% of the input features - there is no error.
When I perform the same dissolve in ArcMap, using either the tool GUI or the python window, it completes fully. So what gives? Is there something I can do to stop the stand alone script finishing early? - it is intended to become a longer script but at the moment it's set up to just perform the dissolve using all default settings.
(Arc 10.2)
Perhaps you are running out of memory, which the ArcMap memory caches to disk but the python script isn't able to.
OK I tried this dissolve a few more times (I did not delete previous attempts but am outputting to a different file each time) -
if I run only the script:
if I also have ArcMap open:
My computer is Intel Xeon E5-2620 2GHz (2 processors)
16 GB RAM
24 cores
This behaviour is rather disturbing to me - it undermines the use of python as a reliable method of geoprocessing - how do I catch these errors?
I thought that one of the advantages of using scripts was that I could do other things while they run in the background.
It may depend on where you are running your scripts. Are you using the python ide that is embedded in ArcMap? That may produce different results that running an ide separately (such as Pyscripter or PythonWin etc). There are more questions on this background vs foreground processing... it almost rivals Define Projection vs Project.
In any event, this works for me.. I never work on, or use in a process, any file loaded in ArcMap. If it has a table, I close it if open. If it is a large file, I remove the layer,then I incrementally save the project... I do python stuff in a separate IDE not the one built into ArcMap or Pro. If I am using ArcMap this entails using python 2.7. Since I have Pro installed, I have python 3.4 installed in which case, I can switch over to Pro and/or python 3.4 and let Arcmap and python 2.7 chug along. I have never had issues, but I have only done dissolved on less than 1Gb, I prefer to tile anything larger either spatially or by attribute and reassemble the bits when a combined file is needed. There are links (which I can't find now) on how to work with large data sets, ArcMap will only use 4 Gb of memory regardless of what you have installed so tiling in some form is often the solution whether using tools or by explicitly splitting your datasets and reassembling. Sorry I can't be more helpful, but that's all I have.
Arecent post, updating an old post suggest ArcGIS Pro, may solve this problem Solution to: Dissolve tool does not fully dissolve medium-size dataset
The Dissolve help topic... Dissolve—Data Management toolbox | ArcGIS for Desktop
Tiling large datasets... Tiled processing of large datasets—Appendices | ArcGIS for Desktop
Now you can Dice it yourself Dice—Data Management toolbox | ArcGIS for Desktop
And the recommendation...don't do anything else, when processing large data set.