nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

What is the solution to the 1,000 record limit for some tools and AGOL processes?

DaveGrolling1

I have two point layers, one has 900 records, and the other has about 11,000. I need to run the 'Find Closest Facility' tool on these data. I have tried to find a solution in ArcMap, ArcGIS Pro, and AGOL, to no avail. Does anyone know how I can run this tool for data greater than 1,000 records? Or provide an alternative function that I'm not privy to?

Already tried hosted feature layer and web feature service type storage when executing the tool in AGOL.

Find more posts tagged with

Accepted answers

All comments

DanPatterson_Retired

It isn't available in Pro... check the docs http://pro.arcgis.com/en/pro-app/tool-reference/appendices/unavailable-tools.htm

cnnlly

Hi Dave,

Do you get an error when you attempt to run the analysis?

If you're performing the analysis in ArcGIS Online, you might be limited by the Max Record Count of the feature service which is set to 1000 by default. It's possible to change this by republishing the layer from ArcMap and changing the figure through Parameters, or by changing it in the REST endpoint: Updating Hosted Feature Services in ArcGIS Online | ArcGIS Blog

I'd recommend trying this with a test layer first to see if the process actually works.

Carmel

DaveGrolling1

Carmel_Connolly: Yes, I get the error when I either run it in ArcGIS Pro or ArcGIS Online. I've played with the max record count issue by changing it when publishing the service and also from the REST endpoint by updating the feature definition. Neither of these modified thresholds worked. My concern is that the tool specifically says, "for records up to 1,000".

DaveGrolling1

@Dan_Patterson: The tool in ArcGIS Pro is called Find Closest Facilities; the tool in ArcGIS Online is called Find Nearest

DanPatterson_Retired

Yes.. in Pro Find Closest Facilities isn't available as per my link. Perhaps ArcMap, but then you mayrun into memory limitations

DaveGrolling1

In ArcMap, you need the Network Analyst extension, which I don't have.

cnnlly

If it states it in the help that it's limited to 1000 features, then I doubt there's an out of the box work around: FindClosestFacilities—Help | ArcGIS Desktop

Have you considered running it in batch? Potentially using model builder/python?

Carmel

AndreasHall_0-1763532081926.png

MattCrowder

dtrolling - A couple questions:

Are you trying to find the closest 1 facility (of 11,000) for each of the 900 incidents or are you trying to find the time/distance/directions to all 11,000 facilities for each 900 incidents?
If just finding the closest 1 facility from each incident, would it be OK to somehow chunk the problem so you iterate through the problem for each incident, just passing in the 100 closest facilities to chose the best 1 from.
What is the extent/density of this problem? For example, are the 11,000 facilities/900 incidents spread throughout a single town, metropolitan area, state, county, or country?
Do you need the route shape and/or directions or do you just need the time/distance to reach the facility?
Do you have any programming experience and if so, in what language?

Matt

DaveGrolling1

Hi Matt,

1) Trying to find the closest 1 facility of 11,000 points for each of the 900 incidents; but the export would include drive time, not just distance;

2) Yes, it would be OK to do that; that solution is the only thing I've come up with. In Pro, you can generate a Near Table, so I was thinking about sub-setting the 11,000 points to just 900 by picking up the closest (in distance) single facility, and then trying to run the Closest Facilities tool in Pro or AGOL;

3) The points cover the entire U.S., including AK, HI, PR;

4) Just time/distance to each closest facility;

5) experience with Python mostly, some SQL and web programming languages.

Dave

KristofferWaage_Beck2

HI Dave.

It seems to me that there are alternative methods to approach the problem.

I assume that you do not have ArcMap (including Network Analyst).

If you do not need the shape of the specific route, then I would recommend using the Origin-Destination tool. It will provide an O-D table with both Drivetime and Length.

BUT: If you did have Network Analyst, then other ways to address the problem could include implementing a Cut-off distance in the analysis settings to drastically reduce the problem that needs to be solved. That might do the trick in itself.

Alternatively you could apply fairly big Service Areas to all Facilities (without overlapping buffers), then use a Spatial Join from your Service Areas towards your incidents. Now you have established the closest facility for every incident. The final step then is to iterate (Python) specific routes, between pairs of Incidents and Facilities.

But the OD-Matrice is my best bet. It is quite fast because is doesn't need to write the shape of every route.

Kind regards

Kristoffer Waage Beck

KristofferWaage_Beck2

Dang.. I just did some further reading on the O-D documentation for ArcGIS Online..

Seems we have same problem at hand. There is a maximum of 5000 Routes.

So we are back at ArcMap and Network Analyst Extension.

But in ArcMap you would have the below settings (and It's fast):

My bad.

Kind regards

Kristoffer

MattCrowder

Given the input size being over 1000 features, you're going to have to chunk up the problem. There are various different ways to do this. Here's one workflow that should be easy to automate and doesn't consume many credits:

Create a near table by running the GP Tool "Generate Near Table" so you can find the 10 closest facilities to each incident:
1. Input Features = your incidents feature class
2. Near Features = your facilities feature class
3. Location = checked
4. Find only closest matches = checked
5. Maximum number of closest matches = 10
6. Method = Geodesic
Create an XY Event Layer on this output table using the NEAR_X and NEAR_Y for the x,y named "ClosestFaciliites"
For each feature in your incidents feature class
1. Select incident feature
2. Select features in ClosestFacilites where IN_FID = incident feature's ObjectID
3. Call Ready-To-Use Services > Logistics > World > OriginDestinationCostMatrix > GenerateOriginDestinationCostMatrix with these parameters
  1. Origins = the incident layer that has the 1 incident selected
  2. Destinations = the ClosestFacilities layer with the 10 closest features for that incident selected
  3. Number of Destinations to Find = 1
  4. Origin destination Line Shape = "Straight Line" (optional, but nice to see)
4. The Output Origin Destination Lines layer that is returned should have 1 row that has the Total Time and Total Distance and the Destination OID that you can use to get the facility that was closest to that origin.
5. Write this information to wherever you're storing the results
6. Iterate to the next incident feature

This workflow could be automated using python (so you don't have to do this process 900 times). However, I'm not a python programmer so can't really help with the automating step of it.

Notes on the above steps:

I specified finding the nearest 10 features. You can reduce/increase this number if you want, but the larger the number, the more credits will be consumed.
I chose to use the OD Cost Matrix Solver instead of the Closest Facility Solver because you just need the time/distance and not the shape or driving directions, so the OD Solver should be faster. Plus, it will use fewer credits.
The credit consumption of running this on the full 900 incidents x 11,000 facilities will be 4.5 credits. This was figured out because you will be doing 900 origins * ((1 origin * 10 destinations) * (1 credit / 2000 OD pairs)) = 4.5 credits.
If you were to use the closest facility solver (which is 0.5 credit per route returned), it would be 450 credits. This is because 900 Incidents x (1 incident x 10 destinations, returning closest 1) * .5 credits = 900 x 1 x 0.5 = 450.

Hope this helps,

Matt

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.