Error 99999 using Clip/Intersect on huge polygon datasets

[MartinScheiber]

Hello everybody,

I´m trying to clip/intersect two huge polygon datasets (500.000 and around 3.500.000 Polygons).
Every time the geoprocess ends with the following error message:

ERROR 999999: Error executing function. Invalid Topology [Topoengine error.] Failed to execute (Clip).

I tried "Repair Geometry" and there are no errors anymore, but it still doesn´t work.

I´m using ArcGIS 10.0 on Windows 7, 64 Bit, 4 Gb RAM. Both files are in File Geodatabases.
Any help would be highly appreciated.

[DarrenWiens2]

Wow, I just googled "arcgis topoengine error" and the proposed fixes are numerous ranging from "it's a bug" to "it's a hardware problem" to "convert to [insert data format here]" to "ArcGIS just doesn't work nicely with some huge datasets".

I suggest trying the easiest first: convert to shapefiles and cross your fingers. Sometimes if your gdb feature classes have fancy geometries they cause problems. Since shapefiles use simpler geometries, sometimes they work when gdbfeature classes don't.

[MathewCoyle]

I'd also recommend changing your environment settings to a larger XY tolerance. I usually set it anywhere from 0.1-5 Meters, but that entirely depends on the resolution you want from the output. Another thing you could try is compacting your FGDB (not compress) sometimes may help clear up phantom problems like this.

[MartinScheiber]

Darren, Mathew,
thanks a lot for your answers. Unfortunately neither converting it to a shapefile nor a larger XY tolerance worked. I guess it´s just too much for ArcGIS to handle.
I think the only solution is to split my dataset up in smaller chunks, do the geoprocessing and put those back together afterwards.

[KenHartling]

Darren, Mathew,
thanks a lot for your answers. Unfortunately neither converting it to a shapefile nor a larger XY tolerance worked. I guess it´s just too much for ArcGIS to handle.
I think the only solution is to split my dataset up in smaller chunks, do the geoprocessing and put those back together afterwards.

Martin, Would I be able to get a copy of the data to give it a try here at ESRI?  If you can share the data and don't have our ftp instructions, please let me know and I'll send you instructions.

Thanks,
Ken Hartling
ESRI Geoprocessing Product Engineer

[MartinScheiber · Accepted solution]

Ken,

sorry but I cannot provide the datasets because of legal regulations.

I can describe them though:
1. A polygon dataset representing areas with slope > 2% (derived from a DEM (25m resolution) using Reclassify and Raster to Polygon)
2. A Polygon dataset representing areas within 15m to any open water (rivers, lakes etc.) created by using the Buffer and then Repair Geometry tools

Both datasets are pretty big, covering the area of Germany (several Million features each).

I have already solved my problem now by simply splitting the process up into 16 subprocesses (states) and merging the results in the end. It took some time, but worked without any problems.
My guess is that it were just too many features for ArcGIS to handle it in one go.

[JamieKass]

I am having problems with an identical error, except I am trying Union a relatively large dataset (tens of thousands of features) in the middle of some Python code. The funny thing is, when I try the same Union in ArcMap, it completes in 2 minutes. What may be causing this fake "Topoengine" error within my code? The call to Union is completely identical within the code to the operation performed in ArcMap, in that the inputs and params are the same. My theory is that the local RAM is spent, and if so there seems to be no good way to refresh the RAM mid-process. Anyone else experience something like this?

[KenHartling]

When you say "in the middle of some Python code" how are you running this code?  From a stand alone script from a command line (or double clicking it?) or from the python window in ArcMap?  The available memory available could be different depending on where you run from and on what platform you run on.  What operating system are you running on and what are the machines hardware specs (CPU, RAM)

See my blog post for a more info and guidance:
http://blogs.esri.com/esri/arcgis/2012/06/15/be-successful-overlaying-large-complex-datasets-in-geop...

The blog also runs through a number of steps to verify your input data doesn't contain issues that also could cause failures.

Another thing to look for is what objects you may be holding onto from steps before the union operation in your python code.  Wherever possible, get rid of objects that may be using memory before running Union.

Please let me know if this info helps you solve the problem or if you need additional help.  If you can't figure it out you could send me your script script and data so I can investigate would be helpful.  I'll send you info on secure ways to do so if it comes to that.

Ken
ESRI
Senior Geoprocessing Product Engineer

[curtvprice]

My theory is that the local RAM is spent, and if so there seems to be no good way to refresh the RAM mid-process. Anyone else experience something like this?

If you are correct, the python garbage collection module may be helpful. Be sure any layers you are done with or in_memory datasets you don't need are deleted before you run the Union. (In Arc 10, you can run arcpy.Delete_management() on layers to free them up.)

import gc
gc.enable()
 

Another issue that may be in play is the scratch and current workspace. If the workspaces you were using when running interactively were different, that may make a difference. For example, .mdb is very size-limited, as you are probably aware.

[JamieKass]

I have done some tests and am convinced it's a memory issue. Would you be able to go into a little more depth into how one would use the gc module in an arcpy workflow? I've done some searching and haven't found any good concrete examples. Is this a recommendable workaround for arcpy scripts? Further, can you explain the benefits of manual garbage collection instead of relying on the automatic gc tasks? Thanks a lot.