Solve is not producing routing when more than 100 incidents are added

ChrisCowin_dhs

Good Afternoon!

So I've had an interesting time trying to convert one of my user's projects from on disk StreetMap Premium and Network Analyst to use our Portals version of both. I'm fairly confident that the Network Analyst and StreetMap Premium are setup on the server correctly because all the routing in the portal works great. However this is my first foray into Network Analyst in many years so that might also be a concern.

The issue I'm running into is for a specific project they have probably 200 ~12k row csv files that need to be looped through, added to the analysis layer, closest facility solved, then output as another csv. My user was nice enough to send me over a trimmed down set of inputs so I only had 1 incident and 15 facilities. My changes to the code worked great, I got all the outputs I was expecting and sent it back over to get incorporated to the project. Unfortunately whenever there are more than those 15 incidents the arcpy.na.solve function does not actually generate the Routes sublayer. At first this sounds like a limit on the NetworkAnalysis Routing Service on ArcGIS Server, but it also doesn't work with 100 rows, I was expecting it to work with up to 1999 rows if that was the case.

Anyway here is the code, there are no errors that are thrown it just does not produce the Routes layer in the Closest Facility and thus when I try and convert it to a table nothing is there.

# Get Current Working Directory
path = os.getcwd()
print('Current Working Directory: ', path)

# Build more paths
na_gdb = os.path.join(path, 'Network Adequacy.gdb')
arcpy.env.workspace = na_gdb

closest_facility = os.path.join(na_gdb, 'Closest Facility')
routes = os.path.join(closest_facility, 'Routes')
incidents = os.path.join(closest_facility, 'Incidents')
facilities = os.path.join(closest_facility, 'Facilities')

# Build intermediary paths
mce_incidents_xy = os.path.join(na_gdb, 'Incidents_XYTableToPoint')
mce_facilities_xy = os.path.join(na_gdb, 'Facilities_XYTableToPoint')

# Build output paths
staging = os.path.join(path, 'Staging Data')
incidents_routing = os.path.join(staging, 'Incidents_Routing_ExportTable.csv')
facilities_routing = os.path.join(staging, 'Facilities_Routing_ExportTable.csv')
all_routing = os.path.join(staging, 'Routes_ExportTable.csv')

# Delete previous versions of files
print('Deleting old directories..')
delete_list = [na_gdb, incidents_routing, facilities_routing, all_routing]
for delete in delete_list:
    if arcpy.Exists(delete):
        arcpy.Delete_management(delete)

# Create geodatabase
print('Create fresh gdb')
arcpy.management.CreateFileGDB(
    path,
    'Network Adequacy.gdb',
)

# Add StreetMap Premium to geodatabase
print('Make Closest Facility Analysis Layer')
arcpy.na.MakeClosestFacilityAnalysisLayer(
    portal,
    closest_facility,
    travel_mode_miles
)
#%%
for spec in spec_list:
    for mce in mce_list:
        # Build output paths
        mce_incidents_csv = os.path.join(path, f'01 Incidents Output/{mce}_Incidents.csv')
        mce_facilities_csv = os.path.join(path, f'02 Facilities Output/{mce}_{spec}_Facilities.csv')
        mce_routing_csv = os.path.join(path, f'03 Routing Output/{mce}_{spec}_Routes.csv')

        # Read incident and facilities csv
        mce_incidents_df = pd.read_csv(mce_incidents_csv, dtype=str)
        mce_facilities_df = pd.read_csv(mce_facilities_csv, dtype=str)

        # Calculate routes
        print(f'routing {len(mce_incidents_df)} incidents to {len(mce_facilities_df)} facilities...')
        if len(mce_facilities_df) == 0:
            # If no facilities:
            print(f'No facilities for {mce} - {spec}')
            incident_column = mce_incidents_df['Incident_Match_ID']
            
            # Build report for routing
            mce_routing_df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': np.nan,
                f'Driving_Minutes': np.nan,
                f'Driving_Miles': np.nan
            })
            
            print('Export No Facilities Report csv')
            mce_routing_df.to_csv(mce_routing_csv, index=False)
        
        else:
            # Convert CSV to GDB Table
            incidents_gdb_table = arcpy.conversion.ExportTable(mce_incidents_csv, os.path.join(na_gdb, 'Incidents_GDB_Table'))
            facilities_gdb_table = arcpy.conversion.ExportTable(mce_facilities_csv, os.path.join(na_gdb, 'Facilities_GDB_Table'))

            # If facilities:
            # Add incident data to geodatabase
            print('Building Incidents XY Table')
            arcpy.management.XYTableToPoint(
                incidents_gdb_table,
                mce_incidents_xy, 
                'X', 
                'Y', 
                None, 
                coordinate_system
            )
        
            # Add facility data to geodatabase
            print('Building Facilities XY Table')
            arcpy.management.XYTableToPoint(
                facilities_gdb_table,
                mce_facilities_xy,
                'X', 
                'Y', 
                None, 
                coordinate_system
            )
        
            # Add incident locations to map
            print('Adding Incidents XY Table to Analysis Layer')
            arcpy.na.AddLocations(
                closest_facility,
                'Incidents', 
                mce_incidents_xy,
                field_mappings_locations,
            )
        
            # Add facility locations to map
            print('Adding Facilities XY Table to Analysis Layer')
            arcpy.na.AddLocations(
                closest_facility,
                'Facilities', 
                mce_facilities_xy,
                field_mappings_locations,
            )
        
            # Run the closest facility tool
            print('Solving')
            arcpy.na.Solve(
                closest_facility,
                'SKIP', 
                'TERMINATE',
            )
        
            # Export Routes table
            print('Exporting Routes')
            arcpy.conversion.ExportTable(
                routes,
                all_routing,
            )
        
            # Export Incidents table
            print('Exporting Incidents Table')
            arcpy.conversion.ExportTable(
                incidents,
                incidents_routing,
            )
        
            # Export Facilities table
            print('Exporting Facilities Table')
            arcpy.conversion.ExportTable(
                facilities,
                facilities_routing,
            )
        
            # Read CSV into dataframes
            print('Building Routing Table')
            routes_df = pd.read_csv(all_routing)
        
            # Set variables for columns
            incident_column = routes_df['IncidentID'] 
            facility_column = routes_df['FacilityID']
            route_column = routes_df['Name']
            minutes_column = routes_df['Total_TravelTime']
            miles_column = routes_df['Total_Miles']
        
            # Create final dataframe
            df = pd.DataFrame({
                'MCE_Name': mce,
                'Specialty': spec,
                'Incident_Match_ID': incident_column,
                'Facility_Match_ID': facility_column,
                'Route': route_column,
                'Driving_Minutes': minutes_column,
                'Driving_Miles': miles_column
            })
        
            # Save as csv
            print(f'Writing {mce} - {spec} to csv..')
            df.to_csv(mce_routing_csv, index=False)

Melinda Morang

It's interesting that you're not getting an errors here.  If the exported Routes sublayer is empty, then I would suspect that the solve is failing, but that would throw an arcpy.executeError.  Try printing out the GP messages after calling arcpy.na.Solve().  arcpy.GetMessages() will return all messages returned by the tool.  Maybe there are some warnings that will explain what's going on.

With that said, if a Python script or process is your ultimate goal, there's a better way to do it than using Make*Layer, Add Locations, Solve in a script.  That workflow works great in the UI, but for Python, there is a much faster and more Pythonic way to code up a network analysis workflow using the arcpy.nax solver objects.  This documentation explains the process: https://pro.arcgis.com/en/pro-app/latest/arcpy/network-analyst/performing-network-analysis.htm.  This documentation is for the ClosestFacility object: https://pro.arcgis.com/en/pro-app/latest/arcpy/network-analyst/closestfacility.htm.  I recommend just switching over your code to use this instead, and maybe your problem will go away.

Another thing you can try (with your existing script or with new code using the arcpy.nax solver objects) is to write the code to use the local Streetmap Premium network dataset and be sure it works.  Once that's working, you should be able to switch the network data source to use the URL to your portal service, and it should just work (provided the portal is signed in).  If it works with local Streetmap Premium and then doesn't work with the portal, then something is wrong with the portal configuration or the log-in.