Intersect polygons and segments does not complete normally

[EricMartin]

I have shapefiles with segments (representing roads, but they're just regular polylines, not a network dataset) and polygons (~14,000 polygons generated with service areas). I'm running intersect to get the segments (and their lengths) within each polygon. On the first run, I got errors, and the tool did not complete. I reduced the segments dataset (from lines representing the whole county to just those within the city) and ran "repair geometry," which did find some problems and fixed them. When I ran it again, the analysis took 12+ hours and completed. The table looked normal, but the resulting featureset was suspicious. It had exactly 64,000 segments in it, and only a few tiny segments mapped. The rest did not appear on the map. Repair geometry on the results featureset had no effect. I ran the analysis again from python. Again it took 12+ hours. This time there were exactly 70,000 segments in the results, and none of them mapped.  So, I don't have confidence that these are complete results. 

I ran a similar analysis with the same polygons shapefile and another set of segments from the same road system; this one had many, many more segments, and the analysis completed normally. It took maybe four hours to run on my university's machine. 

Seems like maybe the analysis is maxing out the computing resources, causing ARCMap to choke and produce bad results. But why could it run the previous, much more complex analysis? Advice? 

[EricMartin]

Thanks, Shantonu. I did not understand what you were suggesting. Though I am using Desktop, it's running remote on a server somewhere. I don't have any control or information about the platform or computing resources for the machine upon which the software is running (without asking the administrators). My university has some pretty impressive research computing resources. I would be very surprised if this service as not adequate for very intensive use.

[EricMartin]

Thanks for your advice! I'm a little foggy on the architecture, since the program is running on a virtual machine somewhere. However, the files are stored on separate server. I have done these big calculations with files all over the place and never had problems like this. However, I'll try moving them to the "machine" where the program is running and see if that helps.

This morning I tried selecting only the first 1000 polygons, and that completed in 20 minutes or so. Extrapolating the results (segments in each polygon) from the first 1000 polygons gives a final result set of "only" ~2.5 million, which should not be a problem for maxing out the file limits, as I understand it.

The second 1000 polygons has been running for hours. Makes me wonder if there is a one or more problem polygon in there not detected by repair geometry.

I appreciate your help!

[DanPatterson_Retired]

I would contact support.  As soon as you indicate you are running arcmap on some virtual machine with data somewhere else, you have introduced two bottlenecks on top of the fact that you are using arcmap with a large dataset and doing an intersect.  You may have reached the limits of your setup.

[DuncanHornby]

Eric,

I've done some hefty union operations in the past (I know this is about intersect but relevant to you).  I found that when you are running such tools, even if "only" a few thousand go in but millions of features come out you are best running this in the 64bit background environment, well it nailed it for me!

But as you have hinted you may have some dodgy geometries:

• If it were me I would be sorting the the polygons by size to see if there are any unusually small polygons.
• Zoom to the extent of the data, is it what you expected?
• Do you have overlapping polygons?
• You say they are simple, get a part count and prove that. Do the same for your road dataset.
• Another thing to check is the number of vertices in each feature, are there unusually large numbers of vertices?
• How are you running this, as a python script, check the geoprocessor environment settings, e.g. some extent setting is not resetting.
• If shapefiles are the input format make sure they have spatial index built and that they are the same coordinate system.
• Check for logical situation that the service area actually intersects your network, from your description it is not possible to conclude that the polygons are generated from the same network.

From the volume of data you are talking it should only take a few hours at most...

Duncan