What is the solution to the 1,000 record limit for some tools and AGOL processes?

[DaveGrolling1]

I have two point layers, one has 900 records, and the other has about 11,000. I need to run the 'Find Closest Facility' tool on these data. I have tried to find a solution in ArcMap, ArcGIS Pro, and AGOL, to no avail. Does anyone know how I can run this tool for data greater than 1,000 records? Or provide an alternative function that I'm not privy to?

Already tried hosted feature layer and web feature service type storage when executing the tool in AGOL.

[MattCrowder]

dtrolling  - A couple questions: 

• Are you trying to find the closest 1 facility (of 11,000) for each of the 900 incidents or are you trying to find the time/distance/directions to all 11,000 facilities for each 900 incidents?
• If just finding the closest 1 facility from each incident, would it be OK to somehow chunk the problem so you iterate through the problem for each incident, just passing in the 100 closest facilities to chose the best 1 from.
• What is the extent/density of this problem?  For example, are the 11,000 facilities/900 incidents spread throughout a single town, metropolitan area, state, county, or country?
• Do you need the route shape and/or directions or do you just need the time/distance to reach the facility?
• Do you have any programming experience and if so, in what language?

Matt

[DaveGrolling1]

Hi Matt,

1) Trying to find the closest 1 facility of 11,000 points for each of the 900 incidents; but the export would include drive time, not just distance;

2) Yes, it would be OK to do that; that solution is the only thing I've come up with. In Pro, you can generate a Near Table, so I was thinking about sub-setting the 11,000 points to just 900 by picking up the closest (in distance) single facility, and then trying to run the Closest Facilities tool in Pro or AGOL;

3) The points cover the entire U.S., including AK, HI, PR;

4) Just time/distance to each closest facility;

5) experience with Python mostly, some SQL and web programming languages.

Dave

[KristofferWaage_Beck2]

HI Dave.

It seems to me that there are alternative methods to approach the problem.

I assume that you do not have ArcMap (including Network Analyst). 

If you do not need the shape of the specific route, then I would recommend using the Origin-Destination tool. It will provide an O-D table with both Drivetime and Length.

BUT: If you did have Network Analyst, then other ways to address the problem could include implementing a Cut-off distance in the analysis settings to drastically reduce the problem that needs to be solved. That might do the trick in itself.

Alternatively you could apply fairly big Service Areas to all Facilities (without overlapping buffers), then use a Spatial Join from your Service Areas towards your incidents. Now you have established the closest facility for every incident. The final step then is to iterate (Python) specific routes, between pairs of Incidents and Facilities.

But the OD-Matrice is my best bet. It is quite fast because is doesn't need to write the shape of every route.

Kind regards

Kristoffer Waage Beck

[KristofferWaage_Beck2]

Dang.. I just did some further reading on the O-D documentation for ArcGIS Online.. 

Seems we have same problem at hand. There is a maximum of 5000 Routes.

So we are back at ArcMap and Network Analyst Extension.

But in ArcMap you would have the below settings (and It's fast):

My bad.

Kind regards

Kristoffer

[MattCrowder]

Given the input size being over 1000 features, you're going to have to chunk up the problem.  There are various different ways to do this.   Here's one workflow that should be easy to automate and doesn't consume many credits:

1. Create a near table by running the GP Tool "Generate Near Table" so you can find the 10 closest facilities to each incident:
   1. Input Features = your incidents feature class
   2. Near Features = your facilities feature class
   3. Location = checked
   4. Find only closest matches = checked
   5. Maximum number of closest matches = 10
   6. Method = Geodesic
2. Create an XY Event Layer on this output table using the NEAR_X and NEAR_Y for the x,y named "ClosestFaciliites"
3. For each feature in your incidents feature class
   1. Select incident feature
   2. Select features in ClosestFacilites where IN_FID = incident feature's ObjectID
   3. Call Ready-To-Use Services > Logistics > World > OriginDestinationCostMatrix > GenerateOriginDestinationCostMatrix with these parameters
      1. Origins = the incident layer that has the 1 incident selected
      2. Destinations = the ClosestFacilities layer with the 10 closest features for that incident selected
      3. Number of Destinations to Find = 1
      4. Origin destination Line Shape = "Straight Line"   (optional, but nice to see)
   4. The Output Origin Destination Lines layer that is returned should have 1 row that has the Total Time and Total Distance and the Destination OID that you can use to get the facility that was closest to that origin.
   5. Write this information to wherever you're storing the results
   6. Iterate to the next incident feature

This workflow could be automated using python (so you don't have to do this process 900 times).  However, I'm not a python programmer so can't really help with the automating step of it.

Notes on the above steps:

1. I specified finding the nearest 10 features.  You can reduce/increase this number if you want, but the larger the number, the more credits will be consumed.
2. I chose to use the OD  Cost Matrix Solver instead of the Closest Facility Solver because you just need the time/distance and not the shape or driving directions, so the OD Solver should be faster.  Plus, it will use fewer credits.
3. The credit consumption of running this on the full 900 incidents x 11,000 facilities will be 4.5 credits.  This was figured out because you will be doing 900 origins * ((1 origin * 10 destinations)  * (1 credit / 2000 OD pairs)) = 4.5 credits.
4. If you were to use the closest facility solver (which is 0.5 credit per route returned), it would be 450 credits.  This is because 900 Incidents x (1 incident x 10 destinations, returning closest 1) * .5 credits = 900 x 1 x 0.5 = 450.

Hope this helps,

Matt