topic Re: Error 99999 using Clip/Intersect on huge polygon datasets in Geoprocessing Questions

Error 99999 using Clip/Intersect on huge polygon datasets

MartinScheiber — Thu, 12 Apr 2012 10:19:58 GMT

Hello everybody,

I´m trying to clip/intersect two huge polygon datasets (500.000 and around 3.500.000 Polygons).
Every time the geoprocess ends with the following error message:

ERROR 999999: Error executing function. Invalid Topology [Topoengine error.] Failed to execute (Clip).

I tried "Repair Geometry" and there are no errors anymore, but it still doesn´t work.

I´m using ArcGIS 10.0 on Windows 7, 64 Bit, 4 Gb RAM. Both files are in File Geodatabases.
Any help would be highly appreciated.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

DarrenWiens2 — Thu, 12 Apr 2012 15:25:32 GMT

Wow, I just googled "arcgis topoengine error" and the proposed fixes are numerous ranging from "it's a bug" to "it's a hardware problem" to "convert to [insert data format here]" to "ArcGIS just doesn't work nicely with some huge datasets".

I suggest trying the easiest first: convert to shapefiles and cross your fingers. Sometimes if your gdb feature classes have fancy geometries they cause problems. Since shapefiles use simpler geometries, sometimes they work when gdbfeature classes don't.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

MathewCoyle — Thu, 12 Apr 2012 18:03:01 GMT

I'd also recommend changing your environment settings to a larger XY tolerance. I usually set it anywhere from 0.1-5 Meters, but that entirely depends on the resolution you want from the output. Another thing you could try is compacting your FGDB (not compress) sometimes may help clear up phantom problems like this.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

MartinScheiber — Fri, 13 Apr 2012 05:21:25 GMT

Darren, Mathew,
thanks a lot for your answers. Unfortunately neither converting it to a shapefile nor a larger XY tolerance worked. I guess it´s just too much for ArcGIS to handle.
I think the only solution is to split my dataset up in smaller chunks, do the geoprocessing and put those back together afterwards.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

KenHartling — Fri, 13 Apr 2012 14:37:09 GMT

Darren, Mathew,
thanks a lot for your answers. Unfortunately neither converting it to a shapefile nor a larger XY tolerance worked. I guess it´s just too much for ArcGIS to handle.
I think the only solution is to split my dataset up in smaller chunks, do the geoprocessing and put those back together afterwards.

Martin, Would I be able to get a copy of the data to give it a try here at ESRI? If you can share the data and don't have our ftp instructions, please let me know and I'll send you instructions.

Thanks,
Ken Hartling
ESRI Geoprocessing Product Engineer

Re: Error 99999 using Clip/Intersect on huge polygon datasets

MartinScheiber — Tue, 17 Apr 2012 11:37:00 GMT

Ken,

sorry but I cannot provide the datasets because of legal regulations.

I can describe them though:
1. A polygon dataset representing areas with slope > 2% (derived from a DEM (25m resolution) using Reclassify and Raster to Polygon)
2. A Polygon dataset representing areas within 15m to any open water (rivers, lakes etc.) created by using the Buffer and then Repair Geometry tools

Both datasets are pretty big, covering the area of Germany (several Million features each).

I have already solved my problem now by simply splitting the process up into 16 subprocesses (states) and merging the results in the end. It took some time, but worked without any problems.
My guess is that it were just too many features for ArcGIS to handle it in one go.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

JamieKass — Fri, 11 Jan 2013 17:02:09 GMT

I am having problems with an identical error, except I am trying Union a relatively large dataset (tens of thousands of features) in the middle of some Python code. The funny thing is, when I try the same Union in ArcMap, it completes in 2 minutes. What may be causing this fake "Topoengine" error within my code? The call to Union is completely identical within the code to the operation performed in ArcMap, in that the inputs and params are the same. My theory is that the local RAM is spent, and if so there seems to be no good way to refresh the RAM mid-process. Anyone else experience something like this?

Re: Error 99999 using Clip/Intersect on huge polygon datasets

KenHartling — Fri, 11 Jan 2013 17:14:57 GMT

When you say "in the middle of some Python code" how are you running this code? From a stand alone script from a command line (or double clicking it?) or from the python window in ArcMap? The available memory available could be different depending on where you run from and on what platform you run on. What operating system are you running on and what are the machines hardware specs (CPU, RAM)

See my blog post for a more info and guidance:
http://blogs.esri.com/esri/arcgis/2012/06/15/be-successful-overlaying-large-complex-datasets-in-geoprocessing/

The blog also runs through a number of steps to verify your input data doesn't contain issues that also could cause failures.

Another thing to look for is what objects you may be holding onto from steps before the union operation in your python code. Wherever possible, get rid of objects that may be using memory before running Union.

Please let me know if this info helps you solve the problem or if you need additional help. If you can't figure it out you could send me your script script and data so I can investigate would be helpful. I'll send you info on secure ways to do so if it comes to that.

Ken
ESRI
Senior Geoprocessing Product Engineer

Re: Error 99999 using Clip/Intersect on huge polygon datasets

curtvprice — Sun, 12 Dec 2021 06:26:33 GMT

My theory is that the local RAM is spent, and if so there seems to be no good way to refresh the RAM mid-process. Anyone else experience something like this?

If you are correct, the python garbage collection module may be helpful. Be sure any layers you are done with or in_memory datasets you don't need are deleted before you run the Union. (In Arc 10, you can run arcpy.Delete_management() on layers to free them up.)

import gc
gc.enable()

Another issue that may be in play is the scratch and current workspace. If the workspaces you were using when running interactively were different, that may make a difference. For example, .mdb is very size-limited, as you are probably aware.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

JamieKass — Mon, 14 Jan 2013 15:46:28 GMT

I have done some tests and am convinced it's a memory issue. Would you be able to go into a little more depth into how one would use the gc module in an arcpy workflow? I've done some searching and haven't found any good concrete examples. Is this a recommendable workaround for arcpy scripts? Further, can you explain the benefits of manual garbage collection instead of relying on the automatic gc tasks? Thanks a lot.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

curtvprice — Mon, 14 Jan 2013 17:09:27 GMT

I have done some tests and am convinced it's a memory issue. Would you be able to go into a little more depth into how one would use the gc module in an arcpy workflow?

I suggest including the above two lines near the top of your script, before you import arcpy, and see if it helps.

The gc module is part of the Python standard library, documented on the python.org website.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

JamieKass — Tue, 15 Jan 2013 18:11:45 GMT

ken2472 -- I am quite sure it is a memory-related issue, as the process runs smoothly and produces output with no geometry errors when processed outside the script in ArcMap. What would my next steps be? I would like to know if there is a good solution to breaking up input polys into smaller pieces for a Union operation, but as each feature is dependent on every other one for a successful and accurate Union, I can't envision a method for subsetting my data.

curtvprice -- I tried some runs after importing gc and enabling it, but I have the same problems. What still confuses me is that I thought gc was automatically enabled anyway. What does enabling gc do then? It seems redundant, unless I do not fully understand the intricacies of gc.

I am still getting ERROR 999999: Error executing function. Invalid Topology [Topoengine error.], Failed to execute (Erase) and Failed to execute (Union). Again, these errors occur on large polygon datasets of approx. 50,000 features and spring up at the end of a long geoprocessing script.

Re: garbage collector

curtvprice — Sun, 12 Dec 2021 06:26:36 GMT

I tried some runs after importing gc and enabling it, but I have the same problems. What still confuses me is that I thought gc was automatically enabled anyway. What does enabling gc do then? It seems redundant, unless I do not fully understand the intricacies of gc.

Now that I re-read the documentation, I think you're absolutely correct.
I thought gc was more than the "generic" garbage collection that Python does, but apparently not.

R:\>C:\Python26\ArcGIS10.0\python.exe
Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import gc
>>> gc.isenabled()
True

Re: Error 99999 using Clip/Intersect on huge polygon datasets

DimitraBouziou — Fri, 13 Dec 2013 20:35:33 GMT

Hello,

I´m also having the same problem (ERROR 999999: Error executing function. Invalid Topology [Topoengine error.] Failed to execute (Intersect)) while trying to intersect a 5,000,000-point feature with a 2,000,000-polygon feature. Based on the discussion above I guess the only way is to break the datasets into smaller parts? I am a new user of ArcGIS and I am not very familiar with the software.

I´m using ArcGIS 10.1 on Windows 7, 64 Bit, 16 Gb RAM. Both files are in a Geodatabase.

Re: Error 99999 using Clip/Intersect on huge polygon datasets

KenHartling — Fri, 13 Dec 2013 20:43:05 GMT

Did you check the blog link that I provided earlier in this thread?

If everything checks out according to the info in that blog, another option is to use the geometry operators. There is a discussion and code sample here.

If it still does not work for you, and if the workaround is not acceptable (the output will be different than the output from the Intersect tool), please submit a technical support incident and provide the data and exact parameters you are using in the Intersect tool so we can investigate.

Thanks,
Ken

Hello,

I´m also having the same problem (ERROR 999999: Error executing function. Invalid Topology [Topoengine error.] Failed to execute (Intersect)) while trying to intersect a 5,000,000-point feature with a 2,000,000-polygon feature. Based on the discussion above I guess the only way is to break the datasets into smaller parts? I am a new user of ArcGIS and I am not very familiar with the software.

I´m using ArcGIS 10.1 on Windows 7, 64 Bit, 16 Gb RAM. Both files are in a Geodatabase.