Working with millions of point features - need to create buffer or select other class by location to subset of the millions of points. Any tips?

Question

Hi everyone,

I need to perform a relatively simple task with an objectively enormous dataset, and my computer doesn't appear to be up to the task. I'm looking for creative solutions.

First, I'm using the "USA Structures" dataset from the Living Atlas. I'm working exclusively with a subset of that data - only residential structures and only within my state. For some reason, I'm not able or not allowed to set up a definition query on the USA Structures dataset, so instead I've had to select by attribute (i.e., type: Residential + state: [my state]). This leaves me with a total of roughly three million polygons selected (since the USA Structures dataset is a polygon feature class).

Next, I need to find out how many points in a separate feature class are within 500 feet of any residential structure from the USA Structures dataset.

I was going to create a 500 foot buffer (dissolved into a single feature) around the residential structures using Pairwise Buffer, but this has failed multiple times at various steps. One time it even completed the buffering part but failed at the "reading features" part of the process... Not that that means anything to me, but it failed at 97% complete instead of 4%, which is to say it was especially heartbreaking. It took almost two hours to get to that point.

So I'm thinking that method is simply not an option. I think the next thing I'll try is selecting by location to see which of the points are within 500 feet of the already selected subset of USA Structures, but I wasn't sure how a selection interacting with a selection would work, especially with a selection of three million features. I'm running that Select by Location option as I type this, but it's not looking good... It's been stuck at 0% for about 30 minutes now.

Anyone have any creative suggestions for me? How would you approach this problem?

I tried moving the subselection of USA Structures into a local file geodatabase using the "Feature to Geodatabase" geoprocessing tool - hoping, perhaps vainly, that it would speed things up - but that one hung up at 0% for more than 30 minutes, and I just gave up. I'm thinking I'll use 911 address points instead of the residential polygons, but this analysis is really specific and I'd love to use the actual shapes of the residential structures as provided in the USA Structures dataset if possible. Some homes are larger than others, and a few dozen feet can make a big difference in what I'm looking at.

Thanks for any help.

TrentonWalker · Answer

Hi @BobBooth1

Thanks for this suggestion. It's a little over my head, but I'll review the links you provided and see if I can work it out. If it does solve my problem, I'll be sure to let you know and mark this as the solution. Thanks again.

BobBooth1 · Answer

I'd suggest subsetting your "other points" dataset first some way (ideally spatially). You could make a fishnet over it, or an array of hexagons:

https://pro.arcgis.com/en/pro-app/3.4/tool-reference/data-management/create-fishnet.htm

Then, in ModelBuilder, iterate over the features in your fishnet, select by location the features in your point feature class (or skip that and spatial join the fishnet polys ID values to your points and iterate by attribute on the fishnet id field).

https://pro.arcgis.com/en/pro-app/3.4/tool-reference/modelbuilder-toolbox/iterate-feature-selection.htm

Use the iteratve output to run the select by distance to house polygons against the LA data and keep track of the resulting selected houses (maybe export them to a new feature class).

DanPatterson · Answer

sorry, I don't work with the Living Atlas

if it is this one, then directing that specific query there may assist you

ArcGIS Living Atlas of the World - Esri Community