Select to view content in your preferred language

Solve is not producing routing when more than 100 incidents are added

414
6
a month ago
ChrisCowin_dhs
New Contributor III

Good Afternoon!

So I've had an interesting time trying to convert one of my user's projects from on disk StreetMap Premium and Network Analyst to use our Portals version of both. I'm fairly confident that the Network Analyst and StreetMap Premium are setup on the server correctly because all the routing in the portal works great. However this is my first foray into Network Analyst in many years so that might also be a concern.

The issue I'm running into is for a specific project they have probably 200 ~12k row csv files that need to be looped through, added to the analysis layer, closest facility solved, then output as another csv. My user was nice enough to send me over a trimmed down set of inputs so I only had 1 incident and 15 facilities. My changes to the code worked great, I got all the outputs I was expecting and sent it back over to get incorporated to the project. Unfortunately whenever there are more than those 15 incidents the arcpy.na.solve function does not actually generate the Routes sublayer. At first this sounds like a limit on the NetworkAnalysis Routing Service on ArcGIS Server, but it also doesn't work with 100 rows, I was expecting it to work with up to 1999 rows if that was the case.

Anyway here is the code, there are no errors that are thrown it just does not produce the Routes layer in the Closest Facility and thus when I try and convert it to a table nothing is there.

# Get Current Working Directory
path = os.getcwd()
print('Current Working Directory: ', path)

# Build more paths
na_gdb = os.path.join(path, 'Network Adequacy.gdb')
arcpy.env.workspace = na_gdb

closest_facility = os.path.join(na_gdb, 'Closest Facility')
routes = os.path.join(closest_facility, 'Routes')
incidents = os.path.join(closest_facility, 'Incidents')
facilities = os.path.join(closest_facility, 'Facilities')

# Build intermediary paths
mce_incidents_xy = os.path.join(na_gdb, 'Incidents_XYTableToPoint')
mce_facilities_xy = os.path.join(na_gdb, 'Facilities_XYTableToPoint')

# Build output paths
staging = os.path.join(path, 'Staging Data')
incidents_routing = os.path.join(staging, 'Incidents_Routing_ExportTable.csv')
facilities_routing = os.path.join(staging, 'Facilities_Routing_ExportTable.csv')
all_routing = os.path.join(staging, 'Routes_ExportTable.csv')

# Delete previous versions of files
print('Deleting old directories..')
delete_list = [na_gdb, incidents_routing, facilities_routing, all_routing]
for delete in delete_list:
    if arcpy.Exists(delete):
        arcpy.Delete_management(delete)

# Create geodatabase
print('Create fresh gdb')
arcpy.management.CreateFileGDB(
    path,
    'Network Adequacy.gdb',
)

# Add StreetMap Premium to geodatabase
print('Make Closest Facility Analysis Layer')
arcpy.na.MakeClosestFacilityAnalysisLayer(
    portal,
    closest_facility,
    travel_mode_miles
)
#%%
for spec in spec_list:
    for mce in mce_list:
        # Build output paths
        mce_incidents_csv = os.path.join(path, f'01 Incidents Output/{mce}_Incidents.csv')
        mce_facilities_csv = os.path.join(path, f'02 Facilities Output/{mce}_{spec}_Facilities.csv')
        mce_routing_csv = os.path.join(path, f'03 Routing Output/{mce}_{spec}_Routes.csv')

        # Read incident and facilities csv
        mce_incidents_df = pd.read_csv(mce_incidents_csv, dtype=str)
        mce_facilities_df = pd.read_csv(mce_facilities_csv, dtype=str)

        # Calculate routes
        print(f'routing {len(mce_incidents_df)} incidents to {len(mce_facilities_df)} facilities...')
        if len(mce_facilities_df) == 0:
            # If no facilities:
            print(f'No facilities for {mce} - {spec}')
            incident_column = mce_incidents_df['Incident_Match_ID']
            
            # Build report for routing
            mce_routing_df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': np.nan,
                f'Driving_Minutes': np.nan,
                f'Driving_Miles': np.nan
            })
            
            print('Export No Facilities Report csv')
            mce_routing_df.to_csv(mce_routing_csv, index=False)
        
        else:
            # Convert CSV to GDB Table
            incidents_gdb_table = arcpy.conversion.ExportTable(mce_incidents_csv, os.path.join(na_gdb, 'Incidents_GDB_Table'))
            facilities_gdb_table = arcpy.conversion.ExportTable(mce_facilities_csv, os.path.join(na_gdb, 'Facilities_GDB_Table'))

            # If facilities:
            # Add incident data to geodatabase
            print('Building Incidents XY Table')
            arcpy.management.XYTableToPoint(
                incidents_gdb_table,
                mce_incidents_xy, 
                'X', 
                'Y', 
                None, 
                coordinate_system
            )
        
            # Add facility data to geodatabase
            print('Building Facilities XY Table')
            arcpy.management.XYTableToPoint(
                facilities_gdb_table,
                mce_facilities_xy,
                'X', 
                'Y', 
                None, 
                coordinate_system
            )
        
            # Add incident locations to map
            print('Adding Incidents XY Table to Analysis Layer')
            arcpy.na.AddLocations(
                closest_facility,
                'Incidents', 
                mce_incidents_xy,
                field_mappings_locations,
            )
        
            # Add facility locations to map
            print('Adding Facilities XY Table to Analysis Layer')
            arcpy.na.AddLocations(
                closest_facility,
                'Facilities', 
                mce_facilities_xy,
                field_mappings_locations,
            )
        
            # Run the closest facility tool
            print('Solving')
            arcpy.na.Solve(
                closest_facility,
                'SKIP', 
                'TERMINATE',
            )
        
            # Export Routes table
            print('Exporting Routes')
            arcpy.conversion.ExportTable(
                routes,
                all_routing,
            )
        
            # Export Incidents table
            print('Exporting Incidents Table')
            arcpy.conversion.ExportTable(
                incidents,
                incidents_routing,
            )
        
            # Export Facilities table
            print('Exporting Facilities Table')
            arcpy.conversion.ExportTable(
                facilities,
                facilities_routing,
            )
        
            # Read CSV into dataframes
            print('Building Routing Table')
            routes_df = pd.read_csv(all_routing)
        
            # Set variables for columns
            incident_column = routes_df['IncidentID'] 
            facility_column = routes_df['FacilityID']
            route_column = routes_df['Name']
            minutes_column = routes_df['Total_TravelTime']
            miles_column = routes_df['Total_Miles']
        
            # Create final dataframe
            df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': facility_column,
                'Route': route_column,
                'Driving_Minutes': minutes_column,
                'Driving_Miles': miles_column
            })
        
            # Save as csv
            print(f'Writing {mce} - {spec} to csv..')
            df.to_csv(mce_routing_csv, index=False)
0 Kudos
6 Replies
MelindaMorang
Esri Regular Contributor

It's interesting that you're not getting an errors here.  If the exported Routes sublayer is empty, then I would suspect that the solve is failing, but that would throw an arcpy.executeError.  Try printing out the GP messages after calling arcpy.na.Solve().  arcpy.GetMessages() will return all messages returned by the tool.  Maybe there are some warnings that will explain what's going on.

With that said, if a Python script or process is your ultimate goal, there's a better way to do it than using Make*Layer, Add Locations, Solve in a script.  That workflow works great in the UI, but for Python, there is a much faster and more Pythonic way to code up a network analysis workflow using the arcpy.nax solver objects.  This documentation explains the process: https://pro.arcgis.com/en/pro-app/latest/arcpy/network-analyst/performing-network-analysis.htm.  This documentation is for the ClosestFacility object: https://pro.arcgis.com/en/pro-app/latest/arcpy/network-analyst/closestfacility.htm.  I recommend just switching over your code to use this instead, and maybe your problem will go away.

Another thing you can try (with your existing script or with new code using the arcpy.nax solver objects) is to write the code to use the local Streetmap Premium network dataset and be sure it works.  Once that's working, you should be able to switch the network data source to use the URL to your portal service, and it should just work (provided the portal is signed in).  If it works with local Streetmap Premium and then doesn't work with the portal, then something is wrong with the portal configuration or the log-in.

 

0 Kudos
ChrisCowin_dhs
New Contributor III

Hey Melinda,

Thanks for the response!
Here is the messages for the solve:

Executing network analysis service Job Id: "jba12b935c62c462fb2cfa398075f8b4b".
Submitted.
Executing...
Network elements with avoid-restrictions are traversed in the output (restriction attribute names: "Avoid Unpaved Roads" "Through Traffic Prohibited").
Succeeded.
Succeeded at Tuesday, June 4, 2024 10:35:27 AM (Elapsed Time: 10.91 seconds)

 

So it certainly thinks its doing everything right! We have no restrictions in the network analysis so I'm assuming we can ignore that line. Will try and convert to the more modern Network Analysis functions and see if that changes anything.

0 Kudos
ChrisCowin_dhs
New Contributor III

A quick side-question, I can't seem to figure out how to create the Network Dataset Layer. All the examples I've seen are pointing to a file location but mine should be pointing to a URL for (I assume) a feature service on the portal correct?

0 Kudos
MelindaMorang
Esri Regular Contributor

Yeah, you can't create a network dataset layer with a URL.  You can just use the URL directly:

arcpy.na.MakeClosestFacilityAnalysisLayer("https://myportal.mydomain.com/portal/")

or

arcpy.nax.ClosetFacility("https://myportal.mydomain.com/portal/")

0 Kudos
ChrisCowin_dhs
New Contributor III

Ah gotcha, that makes sense.

So I have reworked the code to more reflect the the nax documentation provided. I'm getting to the solve and now it is failing:

[0, 'Executing network analysis service Job Id: "jef9f9f2c394d4fb5881f9fafabf52911".'], 
[0, 'Submitted.'], 
[0, 'Executing...'], 
[0, 'No "Facilities" found for "Location 15" in "Incidents".'], 
[0, 'No "Facilities" found for "Location 14" in "Incidents".'],
... # Starts at 15 goes down to 1 then from 16 to 100
[0, 'No "Facilities" found for "Location 100" in "Incidents".'], 
[0, 'ERROR 030212: Solve did not find a solution.'], 
[0, 'No solution found.'], 
[0, 'Failed to execute (FindClosestFacilities).'], 
[0, 'Failed.']

 

Updated Code:

# Get Current Working Directory
path = os.getcwd()
print('Current Working Directory: ', path)

# Build more paths
na_gdb = os.path.join(path, 'Network Adequacy.gdb')
arcpy.env.workspace = na_gdb

closest_facility_path = os.path.join(na_gdb, 'Closest Facility')
routes = os.path.join(closest_facility_path, 'Routes')
incidents = os.path.join(closest_facility_path, 'Incidents')
facilities = os.path.join(closest_facility_path, 'Facilities')

# Build intermediary paths
mce_incidents_xy = os.path.join(na_gdb, 'Incidents_XYTableToPoint')
mce_facilities_xy = os.path.join(na_gdb, 'Facilities_XYTableToPoint')

# Build output paths
staging = os.path.join(path, 'Staging Data')
incidents_routing = os.path.join(staging, 'Incidents_Routing_ExportTable.csv')
facilities_routing = os.path.join(staging, 'Facilities_Routing_ExportTable.csv')
all_routing = os.path.join(staging, 'Routes_ExportTable.csv')

# Delete previous versions of files
print('Deleting old directories..')
delete_list = [na_gdb, incidents_routing, facilities_routing, all_routing]
for delete in delete_list:
    if arcpy.Exists(delete):
        arcpy.Delete_management(delete)

# Create geodatabase
print('Create fresh gdb')
arcpy.management.CreateFileGDB(
    path,
    'Network Adequacy.gdb',
)

#%%
for spec in spec_list:
    for mce in mce_list:
        # Build output paths
        mce_incidents_csv = os.path.join(path, f'01 Incidents Output/{mce}_Incidents.csv')
        mce_facilities_csv = os.path.join(path, f'02 Facilities Output/{mce}_{spec}_Facilities.csv')
        mce_routing_csv = os.path.join(path, f'03 Routing Output/{mce}_{spec}_Routes.csv')

        # Read incident and facilities csv
        mce_incidents_df = pd.read_csv(mce_incidents_csv, dtype=str)
        mce_facilities_df = pd.read_csv(mce_facilities_csv, dtype=str)

        # Calculate routes
        print(f'routing {len(mce_incidents_df)} incidents to {len(mce_facilities_df)} facilities...')
        if len(mce_facilities_df) == 0:
            # If no facilities:
            print(f'No facilities for {mce} - {spec}')
            incident_column = mce_incidents_df['Incident_Match_ID']
            
            # Build report for routing
            mce_routing_df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': np.nan,
                f'Driving_Minutes': np.nan,
                f'Driving_Miles': np.nan
            })
            
            print('Export No Facilities Report csv')
            mce_routing_df.to_csv(mce_routing_csv, index=False)
        
        else:
            arcpy.management.XYTableToPoint(
                path + f'/01 Incidents Output/{mce}_Incidents.csv',
                mce_incidents_xy,
                'X',
                'Y',
                None,
                coordinate_system
            )

            # Add facility data to geodatabase
            arcpy.management.XYTableToPoint(
                path + f'/02 Facilities Output/{mce}_{spec}_Facilities.csv',
                mce_facilities_xy,
                'X',
                'Y',
                None,
                coordinate_system
            )

            # Instantiate a ClosestFacility solver object
            closest_facility = arcpy.nax.ClosestFacility(portal)
            # Set properties
            closest_facility.travelMode = travel_mode
            closest_facility.timeUnits = arcpy.nax.TimeUnits.Minutes
            closest_facility.defaultImpedanceCutoff = 15
            closest_facility.defaultTargetFacilityCount = 1
            closest_facility.routeShapeType = arcpy.nax.RouteShapeType.TrueShapeWithMeasures
            # Load inputs
            closest_facility.load(arcpy.nax.ClosestFacilityInputDataType.Facilities, mce_facilities_xy)
            closest_facility.load(arcpy.nax.ClosestFacilityInputDataType.Incidents, mce_incidents_xy)
            # Solve the analysis
            result = closest_facility.solve()

            # Export the results to a feature class
            if result.solveSucceeded:
                result.export(arcpy.nax.ClosestFacilityOutputDataType.Routes, all_routing)
            else:
                print("Solve failed")
                print(result.solverMessages(arcpy.nax.MessageSeverity.All))

            # Export Routes table
            print('Exporting Routes')
            arcpy.conversion.ExportTable(
                routes,
                all_routing,
            )

            # Export Incidents table
            print('Exporting Incidents Table')
            arcpy.conversion.ExportTable(
                incidents,
                incidents_routing,
            )

            # Export Facilities table
            print('Exporting Facilities Table')
            arcpy.conversion.ExportTable(
                facilities,
                facilities_routing,
            )

            # Read CSV into dataframes
            print('Building Routing Table')
            routes_df = pd.read_csv(all_routing)

            # Set variables for columns
            incident_column = routes_df['IncidentID']
            facility_column = routes_df['FacilityID']
            route_column = routes_df['Name']
            minutes_column = routes_df['Total_TravelTime']
            miles_column = routes_df['Total_Miles']

            # Create final dataframe
            df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': facility_column,
                'Route': route_column,
                'Driving_Minutes': minutes_column,
                'Driving_Miles': miles_column
            })

            # Save as csv
            print(f'Writing {mce} - {spec} to csv..')
            df.to_csv(mce_routing_csv, index=False)

 

So from the error message it seems like the incidents (100) and the facilities (1) are being added, but the routes are failing to generate.

I went and tried to replicate this in the Pro GUI and got the same results, I then tried in the GUI with Output Geometry to Straight Line and it output that fine so that is leading me to believe the Incidents are not being placed on the network. Is there a tolerance setting or should I be doing something else to generate the distance a point is from joining the road network? I remember messing with this about 10 years ago and specifically remember making a line feature class of the closest road network vertex with each facility and incident so that the tolerance could be customized.

ChrisCowin_dhs_0-1717539784563.png

 

0 Kudos
ChrisCowin_dhs
New Contributor III

Quick update I was able to get it to solve for 15 rows with the code above, I needed to raise the impedance cutoff so I just set it to 2000 and it worked great. However when I tried with 100 rows it produced the dreaded

RuntimeError: Result is not available.

So this is happening at line 106 where I'm exporting the results, the solve object says that the solve was successful. So I'm not sure if it is solving correctly and I'm exporting incorrectly (which wouldnt make a ton of sense) or if the RuntimeError is masking the solve failure?

0 Kudos