Working with millions of point features - need to create buffer or select other class by location to subset of the millions of points. Any tips?

[TrentonWalker]

Hi everyone,

I need to perform a relatively simple task with an objectively enormous dataset, and my computer doesn't appear to be up to the task. I'm looking for creative solutions.

First, I'm using the "USA Structures" dataset from the Living Atlas. I'm working exclusively with a subset of that data - only residential structures and only within my state. For some reason, I'm not able or not allowed to set up a definition query on the USA Structures dataset, so instead I've had to select by attribute (i.e., type: Residential + state: [my state]). This leaves me with a total of roughly three million polygons selected (since the USA Structures dataset is a polygon feature class). 

Next, I need to find out how many points in a separate feature class are within 500 feet of any residential structure from the USA Structures dataset.

I was going to create a 500 foot buffer (dissolved into a single feature) around the residential structures using Pairwise Buffer, but this has failed multiple times at various steps. One time it even completed the buffering part but failed at the "reading features" part of the process... Not that that means anything to me, but it failed at 97% complete instead of 4%, which is to say it was especially heartbreaking. It took almost two hours to get to that point.

So I'm thinking that method is simply not an option. I think the next thing I'll try is selecting by location to see which of the points are within 500 feet of the already selected subset of USA Structures, but I wasn't sure how a selection interacting with a selection would work, especially with a selection of three million features. I'm running that Select by Location option as I type this, but it's not looking good... It's been stuck at 0% for about 30 minutes now. 

Anyone have any creative suggestions for me? How would you approach this problem?

I tried moving the subselection of USA Structures into a local file geodatabase using the "Feature to Geodatabase" geoprocessing tool - hoping, perhaps vainly, that it would speed things up - but that one hung up at 0% for more than 30 minutes, and I just gave up. I'm thinking I'll use 911 address points instead of the residential polygons, but this analysis is really specific and I'd love to use the actual shapes of the residential structures as provided in the USA Structures dataset if possible. Some homes are larger than others, and a few dozen feet can make a big difference in what I'm looking at.

Thanks for any help.

[DanPatterson]

if you are getting polygons building, why not simplify the polygons to points (which would be their centroid)

Feature To Point (Data Management)—ArcGIS Pro | Documentation

which would be less onerous buffering than the polygon boundaries.  also your buffer distanc

[TrentonWalker]

Hi @DanPatterson 

Thanks for the suggestion. In this case, I'd really like to keep the structures as polygons to maintain the actual shapes of the buildings. With a threshold as low as 500 feet, 30 or 40 feet here or there can make a pretty big difference. 

I've provided some initial numbers to the people who requested the information, so the project is on a pause for the moment. I think if I've got time in the future, I'll just break it all down county by county so it's easier to handle. It'll take forever to set up, but if that's what I've got to do, then I'll do it.

While I have you here, do you have any idea on why I'm not able to apply a definition query to the Esri "USA Structures" layer from the Living Atlas? When I try to create a new expression (including "Building Occupancy Classification is equal to Residential" and "State is equal to [my state]"), it just says "There is an error with the expression." I've got workaround options, but it would be really convenient if I could get that to work. Thanks.

[DanPatterson]

sorry, I don't work with the Living Atlas

if it is this one, then directing that specific query there may assist you

ArcGIS Living Atlas of the World - Esri Community

[BobBooth1]

I'd suggest subsetting your "other points" dataset first some way (ideally spatially). You could make a fishnet over it, or an array of hexagons:

https://pro.arcgis.com/en/pro-app/3.4/tool-reference/data-management/create-fishnet.htm

Then, in ModelBuilder, iterate over the features in your fishnet, select by location the features in your point feature class (or skip that and spatial join the fishnet polys ID values to your points and iterate by attribute on the fishnet id field).

https://pro.arcgis.com/en/pro-app/3.4/tool-reference/modelbuilder-toolbox/iterate-feature-selection....

Use the iteratve output to run the select by distance to house polygons against the LA data and keep track of the resulting selected houses (maybe export them to a new feature class).

[TrentonWalker]

Hi @BobBooth1 

Thanks for this suggestion. It's a little over my head, but I'll review the links you provided and see if I can work it out. If it does solve my problem, I'll be sure to let you know and mark this as the solution. Thanks again.