topic Speeding up Spatial Select? in Python Questions

Speeding up Spatial Select?

StevenPrager — Tue, 03 Apr 2012 19:15:14 GMT

Hi Everyone,

I am working with a large dataset that I need to cursor through and retain the results of some spatial selects to generate another dataset for use outside of the GIS. The input are the junctions and edges of a road network. The process flow, idea anyway, is to grab an edge, perform a spatial select, figure out what junctions are connected, store this information and move on.

This works very well, but is VERY slow. I have a spatial overlay approach to generate the same data but this is not satisfactory when the data are connected based on vertical elevation values. The code looks something like this:

arcpy.MakeFeatureLayer_management(edgeSource, 'edge_lyr')
arcpy.MakeFeatureLayer_management(junctionSource, 'junc_lyr')

ff = T.time()
edgeList = {}
nodePos = {}
goNogo = {}
edges = arcpy.SearchCursor(edgeSource, "", "")
for edge in edges:
    newID = edge.getValue(edgeIDItem)
    cost = edge.getValue(edgeCostItem)
    expr =  edgeIDItem + " = %s " % newID
    arcpy.SelectLayerByAttribute_management ("edge_lyr", "NEW_SELECTION", expr)
    arcpy.SelectLayerByLocation_management('junc_lyr', 'intersect', 'edge_lyr')

    newEdge = []
    juncs = arcpy.SearchCursor('junc_lyr')
    for junc in juncs:
        jid = junc.getValue(juncIDItem)
        geom = junc.Shape.getPart()
        newEdge.append(jid)
        nodePos[jid] = (geom.X, geom.Y)
        goNogo[jid] = [junc.getValue(nogoIDItem), junc.getValue(nogoDistItem)]
    newEdge.append(cost)
    if len(newEdge) == 3:
        edgeList[newID] = newEdge

Does anyone have any ideas how a process like this might be sped up?

Thanks!

Steve

Re: Speeding up Spatial Select?

MathewCoyle — Sat, 11 Dec 2021 18:19:07 GMT

Instead of doing the double selections, try using a feature layer with a where clause.

So replace this

    arcpy.SelectLayerByAttribute_management ("edge_lyr", "NEW_SELECTION", expr)
    arcpy.SelectLayerByLocation_management('junc_lyr', 'intersect', 'edge_lyr')

With something more like this.

    arcpy.MakeFeatureLayer_management(edgeSource, 'edge_lyr', expr)
    arcpy.SelectLayerByLocation_management('junc_lyr', 'intersect', 'edge_lyr')

Also, having nested cursors and GP tools inside a cursor are generally bad ideas. Is there no way you can avoid processing all this line by line?

Re: Speeding up Spatial Select?

StevenPrager — Sat, 11 Dec 2021 18:19:10 GMT

Instead of doing the double selections, try using a feature layer with a where clause.

So replace this
    arcpy.SelectLayerByAttribute_management ("edge_lyr", "NEW_SELECTION", expr)
    arcpy.SelectLayerByLocation_management('junc_lyr', 'intersect', 'edge_lyr')
With something more like this.
    arcpy.MakeFeatureLayer_management(edgeSource, 'edge_lyr', expr)
    arcpy.SelectLayerByLocation_management('junc_lyr', 'intersect', 'edge_lyr')
Also, having nested cursors and GP tools inside a cursor are generally bad ideas. Is there no way you can avoid processing all this line by line?

Thanks, Mathew, I will definitely try that. Your sense is that it would be faster to make a new feature layer every time using the expression, rather than building the selection off of an existing feature layer. Worth a shot.

As mentioned, I have a technique that makes this infinitely faster using spatial overlay then processing the results. However, it only works when the data are planar embedded. If the data have a Z, I can use a 3d spatial select an junctions in one plane will select only the edges in that plane (e.g., a hwy overpass). In sort, I need the network topology that Arc is storing internally, but I cannot see any way to get that via the Python API...

Again, thanks!

Steve