Buffer Performance on Large Datasets

[LornaMurison]

Hi everyone,
I am writing a script that calculates the different types of land cover present within a certain distance of a stream within a catchment.  The basic steps are as follows:
1. Intersect streams and catchments (output: line)
2. Buffer the streams, dissolving based on the catchment ID
3. Perform a bunch of other dissolves, selects, and intersects to ensure that the buffer doesn't extend beyond the stream's catchment
4. Intersect buffer with land cover
5. Export table.

The script is consistently failing with error 999999 on the buffer step which essentially means that the intersected streams and catchments feature class is too large to buffer.  I tried solving this by creating a fishnet, selecting the catchments whose centroid falls within each fishnet cell and performing all the steps separately for each set of catchments and finally merging together all the tables.  I have gotten to the point of dividing my original catchments layer into 9 separate selections, but still the intersection of catchments and streams is too large to be buffered.
If I have to go loop through the script 9+ times, the processing time is really starting to get out of hand.

I know that buffer can create very large files because of all the vertices that are required, but there must be some way around this.  Has anyone else ever had to deal with this issue?

[markdenil]

try generalizing your streams.

Buffering is a generalization process itself: a large enough buffer will just be just a circle.

Depending on the size of your intended buffer, you can weed out a lot of extra verticies before harming the character of the stream network.

Use the orginal data for your work at later stages, but make the bufers on the generalized lines.

[LornaMurison]

Thanks Mark,
I have tried a the two simplify line generalization techniques with different thresholds and still my buffer is not working.  I'm now looking at individually selecting each catchment through a loop in python.  Not sure how long that kind of thing will take to run though.  Guess I will find out 🙂

[LornaMurison]

Like I thought, calculating land cover for each catchment individually takes a very long time.  Each catchment is taking around 15 seconds.  One of my smallest datasets has over 2k catchments which would mean a processing time of around 9 hours.

[JimW1]

Like I thought, calculating land cover for each catchment individually takes a very long time.  Each catchment is taking around 15 seconds.  One of my smallest datasets has over 2k catchments which would mean a processing time of around 9 hours.

A life time ago I used to do this for Conservation Authorities. In my experience, the 99999 buffer error is related to the stream geometry overlapping itself or self-intersecting. Generalize usually takes these errors out but not always. For example, if you set your remove vertices threshold to say 5m and you have 6m of an arc overlapping itself (think a collapsed V) then the error will remain even after simplifying the line. The buffer is basically collapsing back on to itself and then buffering into infinity on the elbow of the arc.

I would suggest running the stream network through 'repair geometry' first and then make a topology to identify self-intersecting and self-overlapping arcs.

Just an aside b/c I remember from trial and error. Never ever buffer a buffer in model builder. If you need to buffer a buffer then simplify the first buffer. This will also cause that dreaded 999999 error on complex stream systems. The buffer tool makes true arc at corners or buffer ends and then the second buffer tries to do the mathematics of buffering a end-point arc with angles rather than vertices. something like that at least 🙂

Cheers

[LornaMurison]

In my experience, the 99999 buffer error is related to the stream geometry overlapping itself or self-intersecting.
I would suggest running the stream network through 'repair geometry' first and then make a topology to identify self-intersecting and self-overlapping arcs.

Thanks for the tip.  I have added a topology and checked for self intersecting and self overlapping features, it did not find any.  I then repaired the geometry and tried buffering again and it still gives the error 999999.

I am back to the fishnet code which breaks the stream network into 9 pieces.  The first piece works, and it fails on the second.  I'm going to try starting from the third and see if I can find a pattern.

[deleted-user-qQeebl6UEQjh]

We have found 2 options on buffering large water poly datasets.
1. Convert poly to line and attempt the buffer.
2. Cut up offending polys.

For option 2 what we created a model that looped through each individual poly and attempted a buffer and gave us IDs for polys that failed the buffer attempt. We found that there were a handful of offending polygons (usually really complex big lakes/long rivers) that, offten times we only had to cut up once and they then would buffer.