topic Re: Buffer tool not finishing on large feature class in Geoprocessing Questions

Buffer tool not finishing on large feature class

MarcCusumano — Mon, 22 Dec 2014 12:28:27 GMT

I am trying to create a new "Cartway" feature class using the buffer tool on a centerline feature class with ~750,000 features in ArcGIS 10.1 SP1. The tool has basic parameters as inputs, such as "FULL", "ROUND", and "ALL", and background geoprocessing is disabled.

The problem I am having is the tool is stuck at 100% completion and does not finish. However the ArcCatalog.exe process is still working (13% CPU, or 100% / 8 cores). I even let it run over the weekend and it is still not done.

The data and scratch workspace are all on solid state drives (SSD's). There are no "spinners" in this workstation to slow things down, so there should be no bottleneck there. The workstation is a Dell T5500 Precision workstation. The specifications are:

2 x Intel Xeon X5677 (2 physical quad-core CPUs, 8 physical cores total, HT disabled).

20GB RAM

2 x NVIDIA Quadro 600 GPUs

120GB Corsair V12 SSD

120GB Corsair Force GT SSD

Any ideas as to how I can get this to finish?

Re: Buffer tool not finishing on large feature class

DuncanHornby — Mon, 22 Dec 2014 17:13:41 GMT

Sounds like you've hit some "internal" problem with the Buffer tool rather than anything to do with your high spec machine. As a solution could you write a simple model and iterate over say batches of 5,000 features, buffer those and do some final merge?

I had to do this recently as I was trying to buffer 400,000 points, it was bombing out as you described. When I batched the process it ran slower but it did complete.

Re: Buffer tool not finishing on large feature class

MarcCusumano — Mon, 22 Dec 2014 20:12:26 GMT

The process has not "bombed", it is still running; in task manager the ArcCatalog.exe process is using 13% CPU (which is 100% of one core, as it is a single-threaded application. 100% / 8 cores = 13%); also the amount of RAM being used by the process fluctuates rapidly (between 500MB to 650MB).

So it is definitely still working.

Splitting up and then merging is not an option due to the way the buffer tool dissolves intersections to remove overlapping polygons. This has to be done as a single process (we do not have time to manually correct thousands of overlapping polygons).

Re: Buffer tool not finishing on large feature class

AnthonyGiles — Mon, 22 Dec 2014 20:18:33 GMT

Marc,

Have you tried using the old style buffer wizard, I find it out performs the buffer tools in the toolbox.

To add the Buffer Wizard to ArcMap, go to Tools > Customize > Commands tab. In the Categories window on the left, scroll down and select Tools. From the Commands window on the right, select the Buffer Wizard tool, then drag and drop the tool onto any toolbar displayed in the ArcMap window.

Worth a try

Anthony

Re: Buffer tool not finishing on large feature class

XanderBakker — Tue, 23 Dec 2014 03:44:19 GMT

We have encountered this problem too. The way we solved it was to buffer without the dissolve option and afterwards running the dissolve tool. This worked for us.

Re: Buffer tool not finishing on large feature class

MarcCusumano — Wed, 24 Dec 2014 16:09:47 GMT

Thank you for the suggestions. I have tried running the buffer wizard in ArcMap, which ran for over 8 hours. When I came back to the office in the morning, there was an error message saying "ArcMap has crashed".

I then tried running the Buffer tool again in ArcCatalog without the dissolve option checked. This worked, however when attempting to run the Dissolve tool on the output I got an error stating "Invalid Topology". I then tried running the repair geometry tool on the output and re-running the Dissolve tool, but received the same error about invalid topology.

Can someone from esri please help?

Re: Buffer tool not finishing on large feature class

DuncanHornby — Wed, 24 Dec 2014 17:04:57 GMT

Are you buffering features that are creating some insanely large multi-part buffer? May be you could upload an image of what you are trying to buffer? You may find that if you batch up your buffering (as I suggested) you may overcome this problem?

Re: Buffer tool not finishing on large feature class

MarcCusumano — Wed, 24 Dec 2014 18:58:02 GMT

We need to create 25 foot buffers for the entire state of New Jersey for our land base. We are using 2013 TeleAtlas centerlien data as the input. It does not have to be multi-part.

I am open to the idea of doing to it in batches, but would not there then be overlap that would require another dissolve?

Why does it matter how many features are being run through the tool? The software should be able to handle large sets of data; it is almost 2015 and 4TB hard drives are less than $150. I have made posts in the past pleading for esri to rework their core product's most basic tools to bring them into the 21st century but we still continually face these types of issues when working with large datasets..

Re: Buffer tool not finishing on large feature class

DuncanHornby — Thu, 25 Dec 2014 17:24:19 GMT

As I understand it the underlying building blocks of desktop is 32bit. So it does not matter if you have the memory, huge storage capacity and the latest cpu it will never be able to utilises them. Your buffering maybe creating some monster geometry that it cannot handle. So if you batch it up into smaller batches with the extra over heading of running the dissolve tool, yes it will be slower but it should run to completion with you doing some final merge.

As the risk of sounding flippant the time and effort you have spent researching on this forum you could have done it all by now with a fairly simple model or python script?

Re: Buffer tool not finishing on large feature class

MarcCusumano — Fri, 26 Dec 2014 16:25:21 GMT

I tried breaking up the input feature class and running dissolve on each part, then merging back together. This worked (except for one of the parts, which throws error 99999, even after repairing geometry), but now running dissolve on the merged feature class gives an "out of memory" error.

Does anybody else have any suggestions?

Re: Buffer tool not finishing on large feature class

DuncanHornby — Fri, 26 Dec 2014 17:20:00 GMT

So you've managed to narrow it down to a single feature? This is good. You could copy that out into its own feature class and manually chop it up into smaller geometries then run that through the buffer and dissolve tools then merge that back into all your other merged data.

Re: Buffer tool not finishing on large feature class

MarcCusumano — Fri, 26 Dec 2014 17:24:42 GMT

I was able to run the dissolve on 4 / 5 feature classes. The one that fails has 170,000 feature inside of it.

Re: Buffer tool not finishing on large feature class

MarcCusumano — Mon, 29 Dec 2014 13:52:27 GMT

Still struggling with this. Chopping up the feature class did not work, I am still getting topoengine errors and 9999 general function failures. Also this method is creating gaps between feature all over the state, which is unacceptable.

Re: Buffer tool not finishing on large feature class

MarcCusumano — Fri, 02 Jan 2015 15:40:28 GMT

We require support from esri on this issue. Who can I contact?

Re: Buffer tool not finishing on large feature class

TOMKONG — Sat, 03 Jan 2015 22:25:58 GMT

Since the ESRI replaced the data model from Coverage to SHAP to Geodatabase, the ArcGIS software geoprocessing tools can't handle the large dataset, and the error message (memory issue...) is wrong also.

One better solution is to convert the data format to the Coverage format, then running the coverage geoprocessing tool -- the coverage geoprocessing tool can deal with the large dataset.

In order to run coverage tool, you should to have the ArcINFO Workstation installed in your PC, then you can add the Coverage Tools in the Toolbox and to active it to run.

Actually the coverage data model is robust format and much better than SHP & GDB.

Re: Buffer tool not finishing on large feature class

DanPatterson_Retired — Sat, 03 Jan 2015 22:41:15 GMT

Up at the very top of this page is the Support and Services link. If you are the owner of the software, you can report it yourself...if it is a corporate account...someone there (sometimes unknown) can do it for you.

Re: Buffer tool not finishing on large feature class

RichardFairhurst — Sat, 03 Jan 2015 23:32:48 GMT

Did you read this help file?‌ It specifically mentions the Dissolve tool and road casings as an example of what causes the kinds of failures you are experiencing. The Dice tool‌ may be a more effective way of removing the errors you are seeing than the partitioning techniques you are using, since it accounts for vertices counts that create the issues you are seeing.

Also, given that this is a buffer, use of the Cartography generalization tools should be considered acceptable, since these buffers are probably not exact boundaries that represent real legal boundaries or any other real world precise information. Buffering is already a form of generalization, and some additional cartographic generalization may be useful specifically for avoiding the errors you are receiving without necessarily making radical changes to the final output.

For example, Dissolving the lines on name attributes and then using Simplify Line at the start of the process prior to creating buffers could have a significant effect on the vertice count of the buffer output, which is the issue you are running into. At least it should eliminate vertices that contribute no real geometry angles if you use a small tolerance value, but retaining those extraneous vertices increases the likelihood of tool failure. If you try the Simplify Line tool review this help on that topic as well. In addition, or as an alternative, you could also try the Simplify Polygons tool prior to using the Dissolve tool to reduce the vertice count of the buffers. That may eliminate large numbers of very small true curve segments, which may cause tolerance errors when they are being split during the Dissolve process.

In addition to the issues mentioned in the help about shapefiles and personal geodatabase size limitations, if you are outputting to shapefile or personal geodatabase, the vertice counts will explode to convert true curves at end caps into nodes only to dissolve them away, so I hope you are only doing this with a file geodatabase. Reducing vertice counts with the Simplify Lines or Simplify Polygon tools also probably helps eliminate many very short true curve segments created by the Dissolve which may also contribute to the geometry errors being reported if you are using a file geodatabase already. Retaining such very small curves will contribute very little to your buffer data analysis in the end and make any other geoprocessing you may do based on the output problematic.

Re: Buffer tool not finishing on large feature class

MarcCusumano — Tue, 06 Jan 2015 19:58:04 GMT

Thank you for the suggestions everyone, although still no luck. Every avenue ends up with 9999 general function failure, topoengine errors, or the tool simply does not finish after allowing over 3 days of processing time.

I have now merged the entire street centerline feature class into one single feature and am attempting to run the buffer tool on that without the dissolve option checked, although it is still at 0% after a half hour.

If this does not work I will open a ticket with esri.

Re: Buffer tool not finishing on large feature class

TOMKONG — Tue, 06 Jan 2015 22:01:31 GMT

Did you try to use Coverage Tool? It can handle large dataset input. I tried few times to run large dataset input by using Coverage Tool (After installing ArcInfo Worksatation, you can have the Coverage Tool in the Toobox), there are no any problems.

Re: Buffer tool not finishing on large feature class

DaleHoneycutt — Wed, 07 Jan 2015 21:36:53 GMT

This issue is definitely in dissolving the buffers as you've discovered (running buffer w/o dissolve option works). If your input is 750,000 centerlines, that means there are 750,000 polygons, many of them with multiple overlaps. These overlaps count as additional polygons during the dissolve process.

I suggest you first run the Unsplit Line tool on the street centerlines.This should create a centerline feature class with significantly fewer features than the 750,000. Use these unsplit centerlines as input to Buffer.

If you have a 64-bit operating system, install and use 64-bit background processing.

PS: Here's a picture of unsplitting street centerlines on a street-name attribute. This shows the potential reduction in features: