topic Re: Batch Solving Many Routes in ArcGIS Network Analyst Questions

Batch Solving Many Routes

ShaharLevenson — Wed, 14 Mar 2012 13:06:58 GMT

Hello all,

I periodically need to calculate the shortest walking distance for over 1.8 million routes (2 points per route).
With such a large amount of routes I cannot use the desktop ArcGIS - loading that many locations isn't realistic.

What would be the best practice? Creating a python script? Using ArcObjects? Something else?

I have been doing the calculations in smaller pieces (100K, more or less) in version 9.x but now, in version 10, I want to improve the process and do it in one "chunk".

I have access to a strong workstation (20 GB RAM, 24 cores). How can I utilize all the processing potential?

I have no need for hierarchy, restrictions or preserving stop order and all I want is the distance - not the actual routes layer.

Any thoughts?

Thanks,
Shahar.

Re: Batch Solving Many Routes

NaAn — Sat, 11 Dec 2021 06:58:25 GMT

Hi,

There are several ways to do the batch solving, for example, use ArcObject or Python. A simple way is to create a route layer template which includes all your analysis settings, i.e. no hierarchy, uncheck all the restrictions. You can add a pair of origin and destination points into the route layer, solve and get the cost, and delete stops and the route from the layer afterwards. Repeat the procedure over all OD pairs, you will get the cost for all.

Here is a simple example with python assuming you have a text input file with contents like:
ID,X1,Y1,X2,Y2
1,-117.195533,34.057058,-117.197204,34.055463
2,-117.17157,46.736774,-117.261162,47.657173

Open ArcMap, add a route layer MyRouteLayer, open python window and run the following code:

import arcpy, os, sys
arcpy.env.overwriteOutput = True

try:
    costDic1 = {}
    f = open(r"C:\temp\OriginDestinations.txt", 'r+')
    layer1 = "MyRouteLayer" #an open in-memory layer
     
    for line in f:
        twoPs = line.split(",")
 
        # if the line contains number
        if twoPs[0].isdigit():
            OPoint = arcpy.Point()
            DPoint = arcpy.Point()
            pointGeometryList = []
            
            OPoint.X = twoPs[1]
            OPoint.Y = twoPs[2]
            DPoint.X = twoPs[3]
            DPoint.Y = twoPs[4]
            pointGeometry = arcpy.PointGeometry(OPoint)
            pointGeometryList.append(pointGeometry)
            pointGeometry = arcpy.PointGeometry(DPoint)
            pointGeometryList.append(pointGeometry)    
     myPoints = "Mypoints"
     arcpy.CopyFeatures_management(pointGeometryList, myPoints) 
            arcpy.na.AddLocations(layer1,"Stops",myPoints)
            arcpy.Solve_na(layer1)
            
            # get the cost info from the layer
            rows = arcpy.SearchCursor(lyrName + "\\Routes")
            for row in rows:
                costDic1[twoPs[0]] = row.Total_TravelTime
                
            # delete stops and route from the layer
            arcpy.DeleteRows_management(lyrName + "\\Stops")
            arcpy.DeleteRows_management(lyrName + "\\Routes")
    f.close 
    
    # Output the result to a txt file    
    fOut = open(r"C:\temp\Out.txt", 'w')
    for costKey in costDic1.keys():
        line = str(costKey) + "," + str(costDic1[costKey])
        fOut.writelines(line)
    fOut.close

except:
    print "Exception"

You can modify the code to open the template layer on disk and run it outside ArcMap. Hope this helps!

Anna

Re: Batch Solving Many Routes

ShaharLevenson — Sun, 18 Mar 2012 12:17:48 GMT

Hello Anna,

Thank you very much! I changed a few lines in the code and it works. 🙂
I'm learning now how to run it outside of ArcMap, as you suggested.

A question:
Since I have many routes, there may be a case in which a route between two points isn't possible.
How can I make the code "skip" the problematic route instead of stopping with an exception?

Re: Batch Solving Many Routes

NaAn — Sat, 11 Dec 2021 06:58:28 GMT

Hi,

You can throw an exception when no solution is found:

try:
    arcpy.Solve_na(layer1)
except (RuntimeError):
    print "No solution is found for " + str(layer1)

Thanks.

Anna

Re: Batch Solving Many Routes

XiangHe1 — Sat, 11 Dec 2021 06:58:31 GMT

Hi,Anna,

Actually, I have the same problem as described in this thread. The difference is that I developed my code in VBA.
I load points into route layer and get the shortest path in a loop. When the route is not possible, the program just stopped and error messages jump out. How can I skip this problematic route and make the program run into the next iteration? Thank you a lot.

Hi,

You can throw an exception when no solution is found:
try:
    arcpy.Solve_na(layer1)
except (RuntimeError):
    print "No solution is found for " + str(layer1)
Thanks.

Anna

Re: Batch Solving Many Routes

NaAn — Wed, 30 May 2012 16:29:40 GMT

Hi Xiang,

You may use this line to continue the loop in VBA:

On Error Resume Next

Thanks.

Anna

Re: Batch Solving Many Routes

MichaelVorster — Tue, 12 Sep 2017 19:22:36 GMT

Hi Anna

I'm hoping you can help. I am new to python coding and have tried to create a script to solve and display multiple routes between OD pairs simultaneously. In my particular case the origins are also the destinations. The script used is as shown below.

# Name: MakeRouteLayer_MultiRouteWorkflow.py
# Description: Calculate the home-work commutes for cyclists and save
# the output to a feature class
# Requirements: Network Analyst Extension

#Import system modules
import arcpy
from arcpy import env

try:
#Check out the Network Analyst extension license
arcpy.CheckOutExtension("Network")

#Set environment settings
env.overwriteOutput = True

      #Set local variables
      inNetworkDataset = "C:/Users/michael.vorster/Google Drive/Masters/Thesis/Model Info/NMBM Roads/5km       Zone/roads_in_5km_classification/Commuter_Perspective_ND.nd"
      inStops_Home = "C:/Users/michael.vorster/Google Drive/Masters/Thesis/Model Info/Cadastral/ArcGIS/Allotment       Centroids/Allotment_Centroids.shp"
      inStops_Work = "C:/Users/michael.vorster/Google Drive/Masters/Thesis/Model Info/Cadastral/ArcGIS/Allotment       Centroids/Allotment_Centroids.shp"
      outNALayerName = "CycleRoutes"
      outRoutesFC = "C:/Users/michael.vorster/Google Drive/Masters/Thesis/Model Info/Route Data/outRoutes"
      impedanceAttribute = "Commuter"

      #Create a new Route layer. Optimize on Commuter, but compute the
      #distance travelled by accumulating the Length attribute.
      outRouteResultObject = arcpy.na.MakeRouteLayer(inNetworkDataset, outNALayerName, impedanceAttribute,       "FIND_BEST_ORDER", "PRESERVE_BOTH", "", ["Commuter", "Length"], "NO_UTURNS", "", "NO_HIERARCHY",       "", "TRUE_LINES_WITH_MEASURES","")

      #Get the layer object from the result object. The route layer can now be
      #referenced using the layer object.
      outNALayer = outRouteResultObject.getOutput(0)

      #Get the names of all the sublayers within the route layer.
      subLayerNames = arcpy.na.GetNAClassNames(outNALayer)
      #Store the layer names that we will use later
      stopsLayerName = subLayerNames["Stops"]
      routesLayerName = subLayerNames["Routes"]

      #Before loading the commuters' home and work locations as route stops, set
      #up field mapping. Map the "ALLOTMENT" field from the input data to
      #the RouteName property in the Stops sublayer, which ensures that each
      #unique ALLOTMENT will be placed in a separate route. Matching
      #ALLOTMENT from inStops_Home and inStops_Work will end up in the same
      #route.
      fieldMappings = arcpy.na.NAClassFieldMappings(outNALayer, stopsLayerName)
      fieldMappings["RouteName"].mappedFieldName = "ALLOTMENT"

      #Add the commuters' home and work locations as Stops. The same field mapping
      #works for both input feature classes because they both have a field called
      #"ALLOTMENT"
      arcpy.na.AddLocations(outNALayer, stopsLayerName, inStops_Home, fieldMappings, "300 meters", "FID",       [["Commuter_Perspective_ND", "SHAPE"], ["Commuter_Perspective_ND_Junctions", "NONE"]], "", "", "SNAP", "", "",       "")
      arcpy.na.AddLocations(outNALayer, stopsLayerName, inStops_Work, fieldMappings, "300 meters", "FID",       [["Commuter_Perspective_ND", "SHAPE"], ["Commuter_Perspective_ND_Junctions", "NONE"]], "", "", "SNAP", "", "",       "")

#Solve the route layer.
arcpy.na.Solve(outNALayer)

      # Get the output Routes sublayer and save it to a feature class
      RoutesSubLayer = arcpy.mapping.ListLayers(outNALayer, routesLayerName)[0]
      arcpy.management.CopyFeatures(RoutesSubLayer, outRoutesFC)

print "Script completed successfully"

except Exception as e:
      # If an error occurred, print line number and error message
      import traceback, sys
      tb = sys.exc_info()[2]
      print "An error occured on line %i" % tb.tb_lineno
      print str(e)

The script runs through the python window and the route layer is created. The stops are added to the map but there are no routes displayed. There is also no route information in the attribute table. I suspect this has to do with the fact that the origins and destinations are the same points. I understand that the route info from point "A" to point "A" is zero (0) but there should be info for "A to B" and "A to C", "A to D"....etc.

Any help with the above would be greatly appreciated.

Thanks,

Mike

Re: Batch Solving Many Routes

JaySandhu — Tue, 12 Sep 2017 21:24:28 GMT

What are you trying to achieve? Are you trying to get a distance matrix for all points that you have? In that case the route solver is the wrong approach. It will only solve the routes for stop pairs loaded with the same ROUTENAME property, which I think in your case is stop A to stop A and then then Stop B to Stop B, etc. It is not going to solve Stop A to Stop B, unless you load that pair as well. So if you want all pairs, use the OD Cost Matrix and load your locations as origins and destinations and solve. It will solve and give you a complete matrix of travel times. But it only computes a straight line route shape (for speed). And in case you need the actual route geometry of the shortest paths, then use the Closest Facility solver.

Jay Sandhu

Re: Batch Solving Many Routes

MichaelVorster — Wed, 13 Sep 2017 04:57:01 GMT

Hi Jay

Thank you for the response.

I'm actually looking for a combination of the OD Cost Matrix and Closest Facility solvers. The OD Cost Matrix gives me everything I need except the actual routes. The resulting attribute table is perfect and if I could get the actual routes then I would be a very happy man.

I have tried the Closest Facility solver as well but experienced the same problem as with the python code in my original post (i.e. no routes were generated, just points). I suspect this is because the origins and destinations are the same points and are all located on the same layer. I could go through the process of moving each of the points onto a new layer and then solving the routes between a single origin on one layer and multiple destinations on another but this is extremely time consuming, especially if there are lots of points.

Is there any way to use the OD Cost Matrix coding but to change to the output shape from a STRAIGHT_LINE to a TRUE_LINE_WITH_MEASURES?

Regards,

Mike

Re: Batch Solving Many Routes

JaySandhu — Wed, 13 Sep 2017 15:48:47 GMT

Mike,

CF is like OD. One difference is that by default it is set to find ONE closest. You need to change it to find the number of locations you have. It will give you all the routes. Look at the TargetFacilityCount parameter when you are doing the make CF layer.

Closest facility analysis—Help | ArcGIS Desktop

Jay Sandhu

Re: Batch Solving Many Routes

MichaelVorster — Wed, 13 Sep 2017 16:48:00 GMT

Hi Jay

You are an absolute legend. Thanks for the help! I just ran the CF solver and it seems to be working just fine.

Thanks again.

Regards,

Mike

Re: Batch Solving Many Routes

MichaelVorster — Sun, 17 Sep 2017 15:36:29 GMT

Hi Jay

I'm hoping you can help me with something else. I am now trying to calculate the straight line or Euclidean distances between the same points. My reasoning for doing this is so that I can get the ratio of straight line distance to route distance for each of the OD pairs.

To get the Euclidean distances between all OD pairs I was going to use the, "Generate Near Table" tool but for some reason it is not available in my ArcToolbox. Are you aware of another method I can use to get the straight line distances?

Kind regards,

Mike

Re: Batch Solving Many Routes

JaySandhu — Wed, 20 Sep 2017 16:55:45 GMT

Within ArcGIS, the Generate Near Table tool is the best way to compute Euclidean distances. The other way would be to script it up in python. You can project the point data to the right projection and then run the Add XY Coordinates tool. Then with the X,Y coordinates, you will have to compute the straight line distance and write it out.

Jay Sandhu

Re: Batch Solving Many Routes

zhaojunwang — Thu, 19 Oct 2017 19:07:22 GMT

Hi Jay,

Thanks for you replies.

I tried to get the distances for 36 shipping routes (36 origin ports and 36 destinations), I get a reasonable map, but only get one route solved, from port 1 to port 72. Do you know why?

Also, can CF or OD matrix do the same thing for every port pair? instead of the closest one/all the combinations between those ports, but to the designated port within each port pair?

Best,

Zhaojun

Re: Batch Solving Many Routes

JaySandhu — Thu, 19 Oct 2017 21:03:33 GMT

CF and OD return routes to locations within the specified distance cutoff or number specified. They do not honor "pairs". To find routes between pairs of locations, you can either solve each pair independently OR better way is to load them into a single route but set a common ID as the ROUTENAME property. That way you can load in 72 stops as 36 route pairs and solve in one route solver. More details here:

Route analysis—Help | ArcGIS Desktop

Jay Sandhu

Re: Batch Solving Many Routes

zhaojunwang — Sun, 22 Oct 2017 15:55:37 GMT

Thanks Jay,

I tried Newroute, and set common ID for origin ports and destination ports, but the routes solved to be only 1, not 36, do you know the possible reason?

Thanks.

Best,

Zhaojun

Re: Batch Solving Many Routes

zhaojunwang — Sun, 22 Oct 2017 15:56:10 GMT

And there is an error:

Warning: The output geometry for "Location 1 - Location 86" in "Routes" has no M values because the maximum M value is larger than the largest M value that can be supported by the M domain.

Re: Batch Solving Many Routes

FrancisSenyah — Thu, 19 Apr 2018 08:34:14 GMT

Hi Jay, I wonder if you can assist with my use case...I'm actually trying to use the closest facility solver to perform a large calculation, (2.5 million origins to 1 destination initially). I'm running in ArcGIS Pro and using a Network Dataset stored in SQL server. After 5 hours or so it fails with the empty geometry error message below which I presume is related to issues with the SDE Road Network being used. Can you advise how to get around this as the network was built for the whole country and inevitably has some invalid geometries. Perhaps a batch approach is best but I am unsure how to go about this. Any advice would be appreciated. Regards, Francis.

Re: Batch Solving Many Routes

JaySandhu — Thu, 19 Apr 2018 18:29:05 GMT

Francis,

The "operation attempted on empty geometry" error message may not have anything to do with the SDE network dataset. It could be that you have run out of some computer resource (memory or disk space).

In ArcGIS Pro, Closest Facility computes the shortest route and then generates the route geometry to write to the output Routes feature class and also generates the driving directions and writes them to the output directions feature class. It is will generate a lot of data for 2.5 million routes.

Also, for a 2.5 million to 1 case, it will be solving 2.5 million routes. At a minimum, you could flip these so that you have a 1 to 2.5 million case and switch the default "Towards Facility" to "Away from Facilities" to get the same answer but the solver will only need to do ONE route solve.

So let me ask, do you need the output route geometry and directions? Or do you simply need the shortest time/distance between these locations? If you do not need the actual route geometry, then use the Origin-Destination Cost Matrix solver. It will be much quicker with less overhead.

OD cost matrix analysis layer—ArcGIS Pro | ArcGIS Desktop

Jay Sandhu

Re: Batch Solving Many Routes

FrancisSenyah — Fri, 20 Apr 2018 06:57:43 GMT

Hi Jay, Very helpful. Based on this I'm thinking it's a memory issue. I'll switch the solve direction in the 1st instance as the geometry and directions would be helpful for quality assurance more than anything. But if this proves inpractical - the Cost Matrix will also be suitable for us so I will give that a go next. Many Thanks, I'll report back with anything of note.