Solving OD Cost Matrix gets slower and slower in the iteration process

2846
2
08-14-2015 03:41 PM
RezaAmindarbari
New Contributor

I have 7000 points, and for each point I want to create a OD cost matrix of that point to all other points and write the result in separate tables - that is 7000 tables with 6999 records each.

Solving the OD cost matrix is initially fast (takes about 3 seconds) for each origin (to 6999 destinations), but as the loop goes further, it gradually slows down. For example after 100 iterations solving OD matrix takes about a minute, and finally it gets so slow that it is not practical to continue -- for all 7000 points (it probably takes several days).  The solving time doesn't depend on where the origin point is located (in the center or periphery of the network).

Just to figure out what is wrong here, I modified the code such that for every 30 iterations, it redefined the OD Cost Matrix layer, and added the destinations again. This made the solving part as fast as it is in the beginning (about 3 second), but it is not a useful workaround as adding 7000 destination locations itself is a very slow process.

the memory usage doesn't change much, but CPU usages increase as the it loops further. 

should I del something after solving the OD matrix in the loop?

here is my code (i took out the part that redefined the OD Matrix for every 30 iteration, that was just for test):

import arcpy,datetime, sys

start=int(sys.argv[1])

end=int(sys.argv[2])

time_1=datetime.datetime.now().ctime()

##print "start",datetime.datetime.now().ctime()

arcpy.env.overwriteOutput=1

network=r"C:\Users\Reza\Documents\NCSU\2015SUMMER\LUC_Simulation\Datasets\street_centerlines_ND.nd"

inOrgins=r"C:\Users\Reza\Documents\NCSU\2015SUMMER\LUC_Simulation\Parcel_points.shp"

inDestinations=r"C:\Users\Reza\Documents\NCSU\2015SUMMER\LUC_Simulation\Parcel_points.shp"

outLayerFile=r"C:\Users\Reza\Documents\NCSU\2015SUMMER\LUC_Simulation\Parcel_points.lyr"

searchTolerance = "1000 Meters"

arcpy.CheckOutExtension("Network")

g = arcpy.Geometry()

geometryList = arcpy.CopyFeatures_management(inOrgins, g)

##print "object geometry created",datetime.datetime.now().ctime()

outNALayer=arcpy.na.MakeODCostMatrixLayer(network,'OD_layer','Minutes','','','','','','','','No_Lines')

##print "OD matrix created",datetime.datetime.now().ctime()

outNALayer = outNALayer.getOutput(0)

subLayerNames = arcpy.na.GetNAClassNames(outNALayer)

originsLayerName = subLayerNames["Origins"]

destinationsLayerName = subLayerNames["Destinations"]

##arcpy.na.AddLocations(outNALayer, destinationsLayerName, inDestinations,"", searchTolerance)

arcpy.na.AddLocations(outNALayer, destinationsLayerName, geometryList,"", searchTolerance)

##print "Destinations loaded",datetime.datetime.now().ctime()

i=start

for inOrgin in geometryList[start:end]:   

    print "myTable_{0}.dbf".format(i)

    arcpy.na.AddLocations(outNALayer, originsLayerName, inOrgin, "",searchTolerance,"","","",'CLEAR')

    print "origin {0} added".format(i),datetime.datetime.now().ctime()

    arcpy.na.Solve(outNALayer)

    print "solved for origin {0}".format(i),datetime.datetime.now().ctime()

    lines_lyr=arcpy.mapping.ListLayers(outNALayer, 'Lines')[0]

    arcpy.TableToTable_conversion(lines_lyr,r"C:\Users\Reza\Documents\NCSU\2015SUMMER\LUC_Simulation\tables","myTable_{0}.dbf".format(i))

    print "dBase for origin {0} creatd".format(i),datetime.datetime.now().ctime()

    i=i+1

0 Kudos
2 Replies
JoshuaBixby
MVP Esteemed Contributor

From the looks of it, you are running this as a standalone script, which eliminates some of the usual questions I would ask.

Looking over the code, it is pretty straightforward.  You have enough timed print statements that I assume you mean arcpy.na.Solve(outNALayer) is slow when you say "solving the OD cost matrix" is slow, i.e., it isn't one of the other statements in the loop that is slowing the loop down.

If it is the Solve tool that is slowing down after so many tens of iterations, it makes me wonder about whether the CLEAR option of the append parameter in arcpy.na.AddLocations has something going on, maybe not a memory leak but some other bug in memory management.

Given the paths of your datasets, it appears you are taking some summer courses.  I am not sure if you have access to Esri Support as a student where you are attending class; if so, it might be worth opening an Esri Support case to troubleshoot this problem.  The only other thing I can think of at the moment is finding that tipping point where rebuilding the OD cost matrix is faster than iterating over it any more.

0 Kudos
RezaAmindarbari
New Contributor

Thanks Joshua,

Yes, I am running this as a standalone script, and yes, arcpy.na.solve(outNALayer) is the line that gets slow. I think something accumulates on outNALayer.

I am going to open a Esri Support case for this.

0 Kudos