Failed to executed (Identity) Invalid Topology [Topoengine error] HELP PLEASE!

[GrantWest]

ArcGIS for Desktop 10.1 Advanced License (ArcInfo)

I have a model I am trying to run on 1.8 million points and 860 polygons. I am creating a buffer around each point and running an Identity with the buffers and my 860 polygons. Using shapefiles or personal geodatabase feature classes, the model would fail almost immediately during the Buffer tool, giving me a generic error like Error executing function, Failed to execute. I converted to a file geodatabase and ran the model again. This time, the buffer step succeeded in about 20 minutes, but the model failed during the Identity tool after about 5 hours. I got the following error this time:

Executing (Identity): Identity C:\temp\BMI_File.gdb\Buff_5 C:\temp\BMI_File.gdb\Parks_file C:\temp\BMI_File.gdb\Identity_5 ALL # NO_RELATIONSHIPS
Start Time: Mon May 06 15:15:22 2013
Reading Features...
Processing Tiles...
ERROR 999999: Error executing function.
The table was not found.
The table was not found. [Identity_5]
The table was not found.
The table was not found. [Identity_5]
Invalid Topology [Topoengine error.]
Failed to execute (Identity).
Failed at Mon May 06 19:47:15 2013 (Elapsed Time: 4 hours 31 minutes 53 seconds)

I ran this on a Dell Precision 3500 Workstation with 3.07 GHz and 8 GB of RAM.

Is this still a memory issue? Is the topology error something different? Should I add a step to repair geometry after my buffer tool before the Identity?

I've read a lot about these types of geoprocessing failures, but have seen little in the way of work-arounds. I've seen mention of using coverages, specifically in command line Arc/Info. This would require some instruction/learning for me, which is fine, but if there are other suggestions for dealing with this geoprocessing error I would really appreciate it.


Thanks,
Grant West
Research Program Associate I
University of Arkansas Dept. of AEAB

[DaleHoneycutt]

First, have a look at this blog article -- it discusses in detail large overlay (1.8 million polygons + 860 polygons is considered large):
http://blogs.esri.com/esri/arcgis/2012/06/15/be-successful-overlaying-large-complex-datasets-in-geop....

In the blog post, it makes mention of the 64-bit background processing product you can install and use.  This is in full release.  Your version of windows needs to be 64-bit.

Beyond the suggestions that the blog post gives, the first thing I would look at is whether you're running out of disk space on your C: drive.  (Kinda smells like that to me.) 

You don't have to perform a repair geometry on the output of buffer.  I assume you checked/trust the geometry of the other feature class (the one with 830 polygons).

If you still get stuck, we can run it here with your data.

[GrantWest]

Mr. Honeycutt,
   Thanks for your response.  I actually read that blog post yesterday while working through my issues on the shapefile run of the same tool.  I liked your stuff and bookmarked your ESRI profile for future reference.  I ran my tool on my supervisor's workstation, which is a more powerful machine with much more capacity.  When I ran the tool, there was 126 GB of free space on the C:/ drive.  Is there another indicator that would signal I am definitely running out of space in the C: drive? How can I tell that?
    I did check the geometry on my original data before running the buffer, so it isn't that.  I downloaded the Service Pack 1 and the 64-bit Background Processor on my own PC today.  I do have Windows 64-bit.  After installing, I tried to test the background processing speed vs. the foreground processing by just doing the buffer on my points.  In the background, it failed quickly and gave me the error: Error 001143 Background server threw an exception. When I ran the buffer in the foreground, it worked fine in about 24 minutes, as expected.  I'm not sure what I did that seems to have messed up my background processing.  Did I not install the patch correctly?  Are there further configurations I need to set?  Once I figure out how to get my background processing up to the 64-bit, I will load that on my supervisor's workstation again and try the tool again. 

Grant

[DaleHoneycutt]

The only way to tell if you're running out of disk space is to periodically check the available space on the drive while the tool is executing.  128GB at the beginning of the run seems like enough to me.  (1.8 million polygons, guessing ~2k per polygon = ~4GB)

The failure of 64-bit background processing is disturbing.  The fact that Buffer comes back quickly with an error seems like it's a machine/OS configuration issue, not a tool issue (unless you're running Buffer from within a script tool and not passing the layer to the script tool as a parameter).  Do other tools work in 64-bit background?  If not, then there's something subtle about your configuration that's amiss and might take a while to figure out.  I haven't heard of problems with 64-bit background and I did a quick check/search of our support site and didn't find anything there either.  There's not much to the install -- I doubt you did anything incorrect. 

Sigh -- I'm not being much help here.  Let me see if I can get you the ftp site to send your data to us -- my contact for this is out of the office at the moment.

[GrantWest]

Ok.  So, first I tried to do another operation in background on a small set of data.  I did an Identity with 10 polygons as input and my 860 polygons as identity features.  This worked just fine.  Then, I tried to run another buffer on a tiny point file with just 10 points again.  This worked just fine also.  I ran the buffer on my 1.8 million points again, just as I did before my last reply, and it worked swimmingly this time, executing the buffer in just over 5 minutes.  The time difference from when I ran it in the foreground (24 minutes) would indicate it is processing in 64-bit background.  I cannot figure out why I got the error last time.  As far as I can tell, I did nothing different. 

Let me know when you get the FTP site to me.  Unfortunately, I am working with a dummy data set of sensitive location data kept blind by the government agency that collects and manages this data.  I am packaging a model together for the agency to run for me on the real data.  Nonetheless, If I can identify what sorts of shortcomings my system has that are contributing to the failure of the tool, I should be able to identify hangups for them based on their system configurations.  If you are able to be successful, it will be helpful to know what differences we have that might lead to errors. 

Grant

[KenHartling]

Hi Grant,

I'm helping Dale out with your issue.  Could you email me at khartling@esri.com and I'll send you an email with instructions on how to get your dataset and repro case to us.

Thanks,
Ken

Ok.  So, first I tried to do another operation in background on a small set of data.  I did an Identity with 10 polygons as input and my 860 polygons as identity features.  This worked just fine.  Then, I tried to run another buffer on a tiny point file with just 10 points again.  This worked just fine also.  I ran the buffer on my 1.8 million points again, just as I did before my last reply, and it worked swimmingly this time, executing the buffer in just over 5 minutes.  The time difference from when I ran it in the foreground (24 minutes) would indicate it is processing in 64-bit background.  I cannot figure out why I got the error last time.  As far as I can tell, I did nothing different. 

Let me know when you get the FTP site to me.  Unfortunately, I am working with a dummy data set of sensitive location data kept blind by the government agency that collects and manages this data.  I am packaging a model together for the agency to run for me on the real data.  Nonetheless, If I can identify what sorts of shortcomings my system has that are contributing to the failure of the tool, I should be able to identify hangups for them based on their system configurations.  If you are able to be successful, it will be helpful to know what differences we have that might lead to errors. 

Grant

[LiamTurley]

Hello Ken, 

I was wondering if a solution to this issue was ever found? I am encountering the same problem in 2018 and am unable to locate a workaround. 

Thank you, 

Liam 

[ChristinaTardif]

was this issue resolved? I've been getting the same error message.

[LiamTurley]

Was this ever resolved? I am having this error in 2018 and it is preventing me from moving forward on my work