topic Re: Intersect polygons and segments does not complete normally in Geoprocessing Questions

Intersect polygons and segments does not complete normally

EricMartin — Sat, 01 Jul 2017 14:12:48 GMT

I have shapefiles with segments (representing roads, but they're just regular polylines, not a network dataset) and polygons (~14,000 polygons generated with service areas). I'm running intersect to get the segments (and their lengths) within each polygon. On the first run, I got errors, and the tool did not complete. I reduced the segments dataset (from lines representing the whole county to just those within the city) and ran "repair geometry," which did find some problems and fixed them. When I ran it again, the analysis took 12+ hours and completed. The table looked normal, but the resulting featureset was suspicious. It had exactly 64,000 segments in it, and only a few tiny segments mapped. The rest did not appear on the map. Repair geometry on the results featureset had no effect. I ran the analysis again from python. Again it took 12+ hours. This time there were exactly 70,000 segments in the results, and none of them mapped. So, I don't have confidence that these are complete results.

I ran a similar analysis with the same polygons shapefile and another set of segments from the same road system; this one had many, many more segments, and the analysis completed normally. It took maybe four hours to run on my university's machine.

Seems like maybe the analysis is maxing out the computing resources, causing ARCMap to choke and produce bad results. But why could it run the previous, much more complex analysis? Advice?

Re: Intersect polygons and segments does not complete normally

shan_sarkar — Sat, 01 Jul 2017 15:45:50 GMT

Eric,

What version of ArcMap are you running on your machines? Make sure you check out the system requirements for that particular version just to be on the safer side.
Refer:

As per the observation you have posted in your query, it is possible the issue may be with your data. Try to export all the line features in a feature class and execute the Check Geometry—Help | ArcGIS Desktop and then back up your data and execute the Repair Geometry—Help | ArcGIS Desktop .

Once you have completed this try to run the Intersect tool.

I hope this helps

~Shan

Re: Intersect polygons and segments does not complete normally

EricMartin — Sat, 01 Jul 2017 16:04:37 GMT

Thanks for your reply. The version is 10.4.1. It's a university install on university machines run by the research computing people, so I assume it's going to be set up correctly and adequately for most normal and even moderately intensive uses.

I will try your suggestions and report back. Thanks!

Re: Intersect polygons and segments does not complete normally

shan_sarkar — Sat, 01 Jul 2017 16:10:33 GMT

Check out

ArcGIS 10.4.x for Desktop system requirements—System Requirements | ArcGIS Desktop

Can You Run It?

Re: Intersect polygons and segments does not complete normally

DanPatterson_Retired — Sat, 01 Jul 2017 16:32:41 GMT

can you provide some information on the coordinate system of the files... and I am assuming that they are the same

Re: Intersect polygons and segments does not complete normally

EricMartin — Sat, 01 Jul 2017 17:07:34 GMT

Ah! I think they are not the same! I'll project them into the same system and try that.

Re: Intersect polygons and segments does not complete normally

EricMartin — Sun, 02 Jul 2017 11:07:44 GMT

I exported the line features to a new features class and checked the geometry. It was fine. (I had previously run repair on both the segments and the polygons.) I projected the polygons so that shapefile was in the same system as the line segments. As a precaution, I moved both shapefiles into the same file geodatabase.

Again the intersect ran a long time--overnight. This time the results had 2048 features, and all the data in the table was blank. Nothing mapped.

Re: Intersect polygons and segments does not complete normally

EricMartin — Sun, 02 Jul 2017 11:09:48 GMT

Repeating my reply to another user so you'll see it.

I doubt that the polygons are very complex. They are generated with service areas on a mostly sqare street grid.

Again the intersect ran a long time--overnight. This time the results had 2048 features, and all the data in the table was blank. Nothing mapped.

Re: Intersect polygons and segments does not complete normally

DanPatterson_Retired — Sun, 02 Jul 2017 13:05:37 GMT

Eric...

If you created two feature classes in the same coordinate system in a file geodatabase AND you have sufficient memory, Intersect will dice the area automatically

http://desktop.arcgis.com/en/arcmap/latest/tools/supplement/tiled-processing-of-large-datasets.htm

You are either choking on memory, the location of the file (ie not on a local machine) or something else.

Repeating for another overnighter without indicating how big the files are wouldn't be prudent. If tiling the area manually by selecting a smaller area subset of the two for testing, doesn't reveal anything, you might want to contact tech support to see if there is anything that the suggestions and you haven't ruled out.

Re: Intersect polygons and segments does not complete normally

shan_sarkar — Sun, 02 Jul 2017 14:47:03 GMT

I agree with Dan_Patterson in fact that exactly what I had suggested in the first place.

Re: Intersect polygons and segments does not complete normally

EricMartin — Sun, 02 Jul 2017 14:47:16 GMT

Thanks for your advice! I'm a little foggy on the architecture, since the program is running on a virtual machine somewhere. However, the files are stored on separate server. I have done these big calculations with files all over the place and never had problems like this. However, I'll try moving them to the "machine" where the program is running and see if that helps.

This morning I tried selecting only the first 1000 polygons, and that completed in 20 minutes or so. Extrapolating the results (segments in each polygon) from the first 1000 polygons gives a final result set of "only" ~2.5 million, which should not be a problem for maxing out the file limits, as I understand it.

The second 1000 polygons has been running for hours. Makes me wonder if there is a one or more problem polygon in there not detected by repair geometry.

I appreciate your help!

Re: Intersect polygons and segments does not complete normally

EricMartin — Sun, 02 Jul 2017 14:53:55 GMT

Thanks, Shantonu. I did not understand what you were suggesting. Though I am using Desktop, it's running remote on a server somewhere. I don't have any control or information about the platform or computing resources for the machine upon which the software is running (without asking the administrators). My university has some pretty impressive research computing resources. I would be very surprised if this service as not adequate for very intensive use.

Re: Intersect polygons and segments does not complete normally

DanPatterson_Retired — Sun, 02 Jul 2017 16:48:02 GMT

I would contact support. As soon as you indicate you are running arcmap on some virtual machine with data somewhere else, you have introduced two bottlenecks on top of the fact that you are using arcmap with a large dataset and doing an intersect. You may have reached the limits of your setup.

Re: Intersect polygons and segments does not complete normally

DuncanHornby — Wed, 05 Jul 2017 15:47:22 GMT

Eric,

I've done some hefty union operations in the past (I know this is about intersect but relevant to you). I found that when you are running such tools, even if "only" a few thousand go in but millions of features come out you are best running this in the 64bit background environment, well it nailed it for me!

But as you have hinted you may have some dodgy geometries:

If it were me I would be sorting the the polygons by size to see if there are any unusually small polygons.
Zoom to the extent of the data, is it what you expected?
Do you have overlapping polygons?
You say they are simple, get a part count and prove that. Do the same for your road dataset.
Another thing to check is the number of vertices in each feature, are there unusually large numbers of vertices?
How are you running this, as a python script, check the geoprocessor environment settings, e.g. some extent setting is not resetting.
If shapefiles are the input format make sure they have spatial index built and that they are the same coordinate system.
Check for logical situation that the service area actually intersects your network, from your description it is not possible to conclude that the polygons are generated from the same network.

From the volume of data you are talking it should only take a few hours at most...

Duncan