Select to view content in your preferred language

Solve is not producing routing when more than 100 incidents are added

690
9
05-31-2024 04:15 PM
ChrisCowin_dhs
Occasional Contributor

Good Afternoon!

So I've had an interesting time trying to convert one of my user's projects from on disk StreetMap Premium and Network Analyst to use our Portals version of both. I'm fairly confident that the Network Analyst and StreetMap Premium are setup on the server correctly because all the routing in the portal works great. However this is my first foray into Network Analyst in many years so that might also be a concern.

The issue I'm running into is for a specific project they have probably 200 ~12k row csv files that need to be looped through, added to the analysis layer, closest facility solved, then output as another csv. My user was nice enough to send me over a trimmed down set of inputs so I only had 1 incident and 15 facilities. My changes to the code worked great, I got all the outputs I was expecting and sent it back over to get incorporated to the project. Unfortunately whenever there are more than those 15 incidents the arcpy.na.solve function does not actually generate the Routes sublayer. At first this sounds like a limit on the NetworkAnalysis Routing Service on ArcGIS Server, but it also doesn't work with 100 rows, I was expecting it to work with up to 1999 rows if that was the case.

Anyway here is the code, there are no errors that are thrown it just does not produce the Routes layer in the Closest Facility and thus when I try and convert it to a table nothing is there.

# Get Current Working Directory
path = os.getcwd()
print('Current Working Directory: ', path)

# Build more paths
na_gdb = os.path.join(path, 'Network Adequacy.gdb')
arcpy.env.workspace = na_gdb

closest_facility = os.path.join(na_gdb, 'Closest Facility')
routes = os.path.join(closest_facility, 'Routes')
incidents = os.path.join(closest_facility, 'Incidents')
facilities = os.path.join(closest_facility, 'Facilities')

# Build intermediary paths
mce_incidents_xy = os.path.join(na_gdb, 'Incidents_XYTableToPoint')
mce_facilities_xy = os.path.join(na_gdb, 'Facilities_XYTableToPoint')

# Build output paths
staging = os.path.join(path, 'Staging Data')
incidents_routing = os.path.join(staging, 'Incidents_Routing_ExportTable.csv')
facilities_routing = os.path.join(staging, 'Facilities_Routing_ExportTable.csv')
all_routing = os.path.join(staging, 'Routes_ExportTable.csv')

# Delete previous versions of files
print('Deleting old directories..')
delete_list = [na_gdb, incidents_routing, facilities_routing, all_routing]
for delete in delete_list:
    if arcpy.Exists(delete):
        arcpy.Delete_management(delete)

# Create geodatabase
print('Create fresh gdb')
arcpy.management.CreateFileGDB(
    path,
    'Network Adequacy.gdb',
)

# Add StreetMap Premium to geodatabase
print('Make Closest Facility Analysis Layer')
arcpy.na.MakeClosestFacilityAnalysisLayer(
    portal,
    closest_facility,
    travel_mode_miles
)
#%%
for spec in spec_list:
    for mce in mce_list:
        # Build output paths
        mce_incidents_csv = os.path.join(path, f'01 Incidents Output/{mce}_Incidents.csv')
        mce_facilities_csv = os.path.join(path, f'02 Facilities Output/{mce}_{spec}_Facilities.csv')
        mce_routing_csv = os.path.join(path, f'03 Routing Output/{mce}_{spec}_Routes.csv')

        # Read incident and facilities csv
        mce_incidents_df = pd.read_csv(mce_incidents_csv, dtype=str)
        mce_facilities_df = pd.read_csv(mce_facilities_csv, dtype=str)

        # Calculate routes
        print(f'routing {len(mce_incidents_df)} incidents to {len(mce_facilities_df)} facilities...')
        if len(mce_facilities_df) == 0:
            # If no facilities:
            print(f'No facilities for {mce} - {spec}')
            incident_column = mce_incidents_df['Incident_Match_ID']
            
            # Build report for routing
            mce_routing_df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': np.nan,
                f'Driving_Minutes': np.nan,
                f'Driving_Miles': np.nan
            })
            
            print('Export No Facilities Report csv')
            mce_routing_df.to_csv(mce_routing_csv, index=False)
        
        else:
            # Convert CSV to GDB Table
            incidents_gdb_table = arcpy.conversion.ExportTable(mce_incidents_csv, os.path.join(na_gdb, 'Incidents_GDB_Table'))
            facilities_gdb_table = arcpy.conversion.ExportTable(mce_facilities_csv, os.path.join(na_gdb, 'Facilities_GDB_Table'))

            # If facilities:
            # Add incident data to geodatabase
            print('Building Incidents XY Table')
            arcpy.management.XYTableToPoint(
                incidents_gdb_table,
                mce_incidents_xy, 
                'X', 
                'Y', 
                None, 
                coordinate_system
            )
        
            # Add facility data to geodatabase
            print('Building Facilities XY Table')
            arcpy.management.XYTableToPoint(
                facilities_gdb_table,
                mce_facilities_xy,
                'X', 
                'Y', 
                None, 
                coordinate_system
            )
        
            # Add incident locations to map
            print('Adding Incidents XY Table to Analysis Layer')
            arcpy.na.AddLocations(
                closest_facility,
                'Incidents', 
                mce_incidents_xy,
                field_mappings_locations,
            )
        
            # Add facility locations to map
            print('Adding Facilities XY Table to Analysis Layer')
            arcpy.na.AddLocations(
                closest_facility,
                'Facilities', 
                mce_facilities_xy,
                field_mappings_locations,
            )
        
            # Run the closest facility tool
            print('Solving')
            arcpy.na.Solve(
                closest_facility,
                'SKIP', 
                'TERMINATE',
            )
        
            # Export Routes table
            print('Exporting Routes')
            arcpy.conversion.ExportTable(
                routes,
                all_routing,
            )
        
            # Export Incidents table
            print('Exporting Incidents Table')
            arcpy.conversion.ExportTable(
                incidents,
                incidents_routing,
            )
        
            # Export Facilities table
            print('Exporting Facilities Table')
            arcpy.conversion.ExportTable(
                facilities,
                facilities_routing,
            )
        
            # Read CSV into dataframes
            print('Building Routing Table')
            routes_df = pd.read_csv(all_routing)
        
            # Set variables for columns
            incident_column = routes_df['IncidentID'] 
            facility_column = routes_df['FacilityID']
            route_column = routes_df['Name']
            minutes_column = routes_df['Total_TravelTime']
            miles_column = routes_df['Total_Miles']
        
            # Create final dataframe
            df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': facility_column,
                'Route': route_column,
                'Driving_Minutes': minutes_column,
                'Driving_Miles': miles_column
            })
        
            # Save as csv
            print(f'Writing {mce} - {spec} to csv..')
            df.to_csv(mce_routing_csv, index=False)
0 Kudos
9 Replies
MelindaMorang
Esri Regular Contributor

It's interesting that you're not getting an errors here.  If the exported Routes sublayer is empty, then I would suspect that the solve is failing, but that would throw an arcpy.executeError.  Try printing out the GP messages after calling arcpy.na.Solve().  arcpy.GetMessages() will return all messages returned by the tool.  Maybe there are some warnings that will explain what's going on.

With that said, if a Python script or process is your ultimate goal, there's a better way to do it than using Make*Layer, Add Locations, Solve in a script.  That workflow works great in the UI, but for Python, there is a much faster and more Pythonic way to code up a network analysis workflow using the arcpy.nax solver objects.  This documentation explains the process: https://pro.arcgis.com/en/pro-app/latest/arcpy/network-analyst/performing-network-analysis.htm.  This documentation is for the ClosestFacility object: https://pro.arcgis.com/en/pro-app/latest/arcpy/network-analyst/closestfacility.htm.  I recommend just switching over your code to use this instead, and maybe your problem will go away.

Another thing you can try (with your existing script or with new code using the arcpy.nax solver objects) is to write the code to use the local Streetmap Premium network dataset and be sure it works.  Once that's working, you should be able to switch the network data source to use the URL to your portal service, and it should just work (provided the portal is signed in).  If it works with local Streetmap Premium and then doesn't work with the portal, then something is wrong with the portal configuration or the log-in.

 

0 Kudos
ChrisCowin_dhs
Occasional Contributor

Hey Melinda,

Thanks for the response!
Here is the messages for the solve:

Executing network analysis service Job Id: "jba12b935c62c462fb2cfa398075f8b4b".
Submitted.
Executing...
Network elements with avoid-restrictions are traversed in the output (restriction attribute names: "Avoid Unpaved Roads" "Through Traffic Prohibited").
Succeeded.
Succeeded at Tuesday, June 4, 2024 10:35:27 AM (Elapsed Time: 10.91 seconds)

 

So it certainly thinks its doing everything right! We have no restrictions in the network analysis so I'm assuming we can ignore that line. Will try and convert to the more modern Network Analysis functions and see if that changes anything.

0 Kudos
ChrisCowin_dhs
Occasional Contributor

A quick side-question, I can't seem to figure out how to create the Network Dataset Layer. All the examples I've seen are pointing to a file location but mine should be pointing to a URL for (I assume) a feature service on the portal correct?

0 Kudos
MelindaMorang
Esri Regular Contributor

Yeah, you can't create a network dataset layer with a URL.  You can just use the URL directly:

arcpy.na.MakeClosestFacilityAnalysisLayer("https://myportal.mydomain.com/portal/")

or

arcpy.nax.ClosetFacility("https://myportal.mydomain.com/portal/")

0 Kudos
ChrisCowin_dhs
Occasional Contributor

Ah gotcha, that makes sense.

So I have reworked the code to more reflect the the nax documentation provided. I'm getting to the solve and now it is failing:

[0, 'Executing network analysis service Job Id: "jef9f9f2c394d4fb5881f9fafabf52911".'], 
[0, 'Submitted.'], 
[0, 'Executing...'], 
[0, 'No "Facilities" found for "Location 15" in "Incidents".'], 
[0, 'No "Facilities" found for "Location 14" in "Incidents".'],
... # Starts at 15 goes down to 1 then from 16 to 100
[0, 'No "Facilities" found for "Location 100" in "Incidents".'], 
[0, 'ERROR 030212: Solve did not find a solution.'], 
[0, 'No solution found.'], 
[0, 'Failed to execute (FindClosestFacilities).'], 
[0, 'Failed.']

 

Updated Code:

# Get Current Working Directory
path = os.getcwd()
print('Current Working Directory: ', path)

# Build more paths
na_gdb = os.path.join(path, 'Network Adequacy.gdb')
arcpy.env.workspace = na_gdb

closest_facility_path = os.path.join(na_gdb, 'Closest Facility')
routes = os.path.join(closest_facility_path, 'Routes')
incidents = os.path.join(closest_facility_path, 'Incidents')
facilities = os.path.join(closest_facility_path, 'Facilities')

# Build intermediary paths
mce_incidents_xy = os.path.join(na_gdb, 'Incidents_XYTableToPoint')
mce_facilities_xy = os.path.join(na_gdb, 'Facilities_XYTableToPoint')

# Build output paths
staging = os.path.join(path, 'Staging Data')
incidents_routing = os.path.join(staging, 'Incidents_Routing_ExportTable.csv')
facilities_routing = os.path.join(staging, 'Facilities_Routing_ExportTable.csv')
all_routing = os.path.join(staging, 'Routes_ExportTable.csv')

# Delete previous versions of files
print('Deleting old directories..')
delete_list = [na_gdb, incidents_routing, facilities_routing, all_routing]
for delete in delete_list:
    if arcpy.Exists(delete):
        arcpy.Delete_management(delete)

# Create geodatabase
print('Create fresh gdb')
arcpy.management.CreateFileGDB(
    path,
    'Network Adequacy.gdb',
)

#%%
for spec in spec_list:
    for mce in mce_list:
        # Build output paths
        mce_incidents_csv = os.path.join(path, f'01 Incidents Output/{mce}_Incidents.csv')
        mce_facilities_csv = os.path.join(path, f'02 Facilities Output/{mce}_{spec}_Facilities.csv')
        mce_routing_csv = os.path.join(path, f'03 Routing Output/{mce}_{spec}_Routes.csv')

        # Read incident and facilities csv
        mce_incidents_df = pd.read_csv(mce_incidents_csv, dtype=str)
        mce_facilities_df = pd.read_csv(mce_facilities_csv, dtype=str)

        # Calculate routes
        print(f'routing {len(mce_incidents_df)} incidents to {len(mce_facilities_df)} facilities...')
        if len(mce_facilities_df) == 0:
            # If no facilities:
            print(f'No facilities for {mce} - {spec}')
            incident_column = mce_incidents_df['Incident_Match_ID']
            
            # Build report for routing
            mce_routing_df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': np.nan,
                f'Driving_Minutes': np.nan,
                f'Driving_Miles': np.nan
            })
            
            print('Export No Facilities Report csv')
            mce_routing_df.to_csv(mce_routing_csv, index=False)
        
        else:
            arcpy.management.XYTableToPoint(
                path + f'/01 Incidents Output/{mce}_Incidents.csv',
                mce_incidents_xy,
                'X',
                'Y',
                None,
                coordinate_system
            )

            # Add facility data to geodatabase
            arcpy.management.XYTableToPoint(
                path + f'/02 Facilities Output/{mce}_{spec}_Facilities.csv',
                mce_facilities_xy,
                'X',
                'Y',
                None,
                coordinate_system
            )

            # Instantiate a ClosestFacility solver object
            closest_facility = arcpy.nax.ClosestFacility(portal)
            # Set properties
            closest_facility.travelMode = travel_mode
            closest_facility.timeUnits = arcpy.nax.TimeUnits.Minutes
            closest_facility.defaultImpedanceCutoff = 15
            closest_facility.defaultTargetFacilityCount = 1
            closest_facility.routeShapeType = arcpy.nax.RouteShapeType.TrueShapeWithMeasures
            # Load inputs
            closest_facility.load(arcpy.nax.ClosestFacilityInputDataType.Facilities, mce_facilities_xy)
            closest_facility.load(arcpy.nax.ClosestFacilityInputDataType.Incidents, mce_incidents_xy)
            # Solve the analysis
            result = closest_facility.solve()

            # Export the results to a feature class
            if result.solveSucceeded:
                result.export(arcpy.nax.ClosestFacilityOutputDataType.Routes, all_routing)
            else:
                print("Solve failed")
                print(result.solverMessages(arcpy.nax.MessageSeverity.All))

            # Export Routes table
            print('Exporting Routes')
            arcpy.conversion.ExportTable(
                routes,
                all_routing,
            )

            # Export Incidents table
            print('Exporting Incidents Table')
            arcpy.conversion.ExportTable(
                incidents,
                incidents_routing,
            )

            # Export Facilities table
            print('Exporting Facilities Table')
            arcpy.conversion.ExportTable(
                facilities,
                facilities_routing,
            )

            # Read CSV into dataframes
            print('Building Routing Table')
            routes_df = pd.read_csv(all_routing)

            # Set variables for columns
            incident_column = routes_df['IncidentID']
            facility_column = routes_df['FacilityID']
            route_column = routes_df['Name']
            minutes_column = routes_df['Total_TravelTime']
            miles_column = routes_df['Total_Miles']

            # Create final dataframe
            df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': facility_column,
                'Route': route_column,
                'Driving_Minutes': minutes_column,
                'Driving_Miles': miles_column
            })

            # Save as csv
            print(f'Writing {mce} - {spec} to csv..')
            df.to_csv(mce_routing_csv, index=False)

 

So from the error message it seems like the incidents (100) and the facilities (1) are being added, but the routes are failing to generate.

I went and tried to replicate this in the Pro GUI and got the same results, I then tried in the GUI with Output Geometry to Straight Line and it output that fine so that is leading me to believe the Incidents are not being placed on the network. Is there a tolerance setting or should I be doing something else to generate the distance a point is from joining the road network? I remember messing with this about 10 years ago and specifically remember making a line feature class of the closest road network vertex with each facility and incident so that the tolerance could be customized.

ChrisCowin_dhs_0-1717539784563.png

 

0 Kudos
ChrisCowin_dhs
Occasional Contributor

Quick update I was able to get it to solve for 15 rows with the code above, I needed to raise the impedance cutoff so I just set it to 2000 and it worked great. However when I tried with 100 rows it produced the dreaded

RuntimeError: Result is not available.

So this is happening at line 106 where I'm exporting the results, the solve object says that the solve was successful. So I'm not sure if it is solving correctly and I'm exporting incorrectly (which wouldnt make a ton of sense) or if the RuntimeError is masking the solve failure?

0 Kudos
ChrisCowin_dhs
Occasional Contributor

Just closing the loop here, have worked extensively trying to get this to work, there were existing licensing issues that needed to be resolved, a memory upgrade and republishing all the routing geoprocessing services and slowly stripping back other things that are interacting with it so even just using the NA REST endpoints with my previous data converted to JSON still does produce an error so there is something fundamentally wrong with the install or the server and is probably not replicable by anyone.

However, if anyone comes across this in the future and is having trouble with a StreetMap Premium producing routes:

1. Turn on debug logging

2. Run Analysis in Pro (like an end user)

3. If still failing and nothing in the logs; try in Portal with the same data but uploaded.

4. If still failing and nothing in the logs; access the developer tools in your browser and you can extract the json of the data from the network tool under submitJob and you can just place that in the html REST endpoint located here: https://machine.domain.com/webAdapter/rest/services/Routing/NetworkAnalysis/GPServer/FindClosestFaci...

Then at least you know nothing else is getting in the way and the entire thing is borked.

ChrisCowin_dhs_0-1724109758393.png

0 Kudos
MelindaMorang
Esri Regular Contributor

Did you actually manage to get it working?  I'm sorry this has proven so frustrating!  At this point, if it's still not working, I suggest you call Esri Support so one of the analysts can work through it with you and try to pin down the source of the problem.  It doesn't seem like the type of problem we can solve here on the forums.

0 Kudos
ChrisCowin_dhs
Occasional Contributor

Unfortunately no, the routing JSON in the REST API would return null even though every logging message says there should be routes. I have been working with several ESRI folks, but unfortunately nobody could figure it out and we have another server that is on 11.3 so we are just going to reinstall it on the other server and my understanding is that the install process on 11.3 has been significantly simplified.

0 Kudos