Select to view content in your preferred language

arcpy.da.SearchCursor skipping rows

491
11
09-13-2024 07:22 AM
jacob_loughridge
Emerging Contributor

I have a script that is using the OID@ as the cursor to iterate through a feature class. It seems to be skipping 3 OIDs at random intervals during the script when using UpdateCursor, but seems to have a consistent 3 row skip when using SearchCursor.

Is there a reason as to why I am getting a random skip here?

 

with arcpy.da.SearchCursor(IterateLayer, ["OID@"]) as cursor:
        for row in cursor:
            rowID = str(row[0])
            arcpy.AddMessage(fr"Processing for Pole: "+ rowID)
           
            PoleSelected = mgt.SelectLayerByAttribute(IterateLayer, "NEW_SELECTION", "OBJECTID = {}".format(row[0]))

            arcpy.AddMessage("Closest River/Stream Processing")

            def Closest_RiverStream(SaR, PoleS):  

                def RiverStream_ID(PoleS):
                    with arcpy.da.SearchCursor((PoleS), "Stream_ID") as cursor:
                        for row in cursor:
                            StreamID = '{}'.format(row[0])
                            StreamID = str(StreamID)
                            return StreamID

                RID = RiverStream_ID(PoleS)
                RID = str(RID)
                clause = "OBJECTID = " + RID
                StreamSelected = mgt.SelectLayerByAttribute(SaR, "NEW_SELECTION", where_clause=clause)

                def StreamPermanence(S_S):
                    with arcpy.da.SearchCursor(S_S, "StreamPermanence"):
                        for row in cursor:
                            StreamPerm = '{}'.format(row[0])
                            StreamPerm = str(StreamPerm)
                            return StreamPerm
                       
                if (RID != "-1"):
                    ParclOwner = mgt.CalculateField(PoleS, "StreamPermanence", "'"+str(StreamPermanence(StreamSelected))+"'")
                else:
                    NotInSearchDist = "Not Within 100 Feet of Stream or River"
                    ParclOwner = mgt.CalculateField(PoleS, "StreamPermanence", "'"+str(NotInSearchDist)+"'")

                def StreamSize(S_S):
                    with arcpy.da.SearchCursor(S_S, "FPAStreamSize"):
                        for row in cursor:
                            Stream_Size = '{}'.format(row[0])
                            Stream_Size = str(Stream_Size)
                            return Stream_Size
                       
                if (RID != "-1"):
                    ParclOwner = mgt.CalculateField(PoleS, "StreamSize", "'"+str(StreamSize(StreamSelected))+"'")
                else:
                    NotInSearchDist = "Not Within 100 Feet of Stream or River"
                    ParclOwner = mgt.CalculateField(PoleS, "StreamSize", "'"+str(NotInSearchDist)+"'")
               
                #mgt.Delete(TempPole)              

            Closest_RiverStream(StreamAndRivers, PoleSelected)

            mgt.SelectLayerByAttribute(StreamAndRivers, "CLEAR_SELECTION")

 

 

0 Kudos
11 Replies
VinceAngelo
Esri Esteemed Contributor

Never ever change the selection environment on a source while inside a cursor using that source.

Please use code formatting on your post so that the code indentation is legible.

- V

jacob_loughridge
Emerging Contributor

I am not sure I follow, are you referencing the inital select by Attribute? 

I have that part in a similar script and it doesn't skip around like this function. I added this function to the larger script and that is when the skipping began, as soon as I removed this function, the skipping stopped in the other script. 

 

0 Kudos
RPGIS
by
Frequent Contributor

So, there are a couple of things to mention, not so much the question but issues that I see with the script itself.

  1. Avoid nesting loops. In terms of performance, nested loops can be processing killers if they are used within themselves.
  2. All cursor functions have SQL clauses that can be used, so having separate select by attribute query layer is unnecessary.
  3. In my personal opinion, never set functions within the main portion of a script. I typically set them above the main processes and below the imports.

 I would also recommend checking out arcpy geometry to get a better understanding of how to utilize feature geometries.

 

import arcpy
from arcpy import ListFields
from arcpy.da import SearchCursor as Searching , UpdateCursor as Updating

# Retrieves records of an input layer/feature and returns a dictionary of values
def GetRecords( Layer ,  ["OID@"] + Fields + ['SHAPE@'] , SQLClause ):
    SearchLayer = Searching( Layer , Fields )
    if SQLClause is not None: SearchLayer = Searching( Layer , Fields , SQLClause )
    return { row[0] : row[ 1:] for row in SearchLayer }

# Returns a list of field names
def GetFieldNames( Layer , ExcludeFields ):
    fieldnames = [ field.name for field in ListFields(  Layer ) ]
    if type( ExcludeFields ) is list: fieldnames = list( set( fieldnames ).difference( set( ExcludeFields ) ) )
    return fieldnames

# Checks the distance between two geometries
def CheckProximity( InputFeatureRecords , DistanceFeatureRecords , SetDistance ):
    WithinSetLimits = { }
    for IF_id , IF_Attributes in InputFeatureRecords.items():
        for DF_id , DF_Attributes in DistanceFeatureRecords.items():
            Point = IF_Attributes[ 0 ]
            Line = DF_Attributes[ 0 ]
            if Point.distanceTo( Line ) <= SetDistance: WithinSetLimits[ IF_id ] = DF_Attributes[ 0 ]
            else: WithinSetLimits[ IF_id ] = "Not Within 100 Feet of Stream or River"
    return WithinSetLimits
        
Poles = '<PoleLayer>'
Streams = '<StreamLayer>'

PoleFields =  GetFieldNames( infeature , '<[ list of field names to exclude ]>' )
StreamFields = [ "Stream_ID" , "StreamPermanence" ]

PoleRecords = GetRecords( infeature ,  ["OID@"] + Fields + ['SHAPE@'] )
StreamRecords = GetRecords( infeature ,  ["OID@"] + Fields + ['SHAPE@'] )

CheckProximity( PoleRecords , StreamRecords , 100 )

 

The sample above isn't a solution but I just thought to provide something useful to give you a rough idea.

jacob_loughridge
Emerging Contributor

Thanks, for the reply.

this portion is replacing the calculate field i have running in my script now correct?

            if Point.distanceTo( Line ) <= SetDistance: WithinSetLimits[ IF_id ] = DF_Attributes[ 0 ]
            else: WithinSetLimits[ IF_id ] = "Not Within 100 Feet of Stream or River"

 

RPGIS
by
Frequent Contributor

I rewrote your script as such because it was really difficult to read and identify where the issue was.

Have you checked out the link in my previous post? It shows you how to utilize the geometry of a feature.

The line of code you are trying to use, are you merely copy and paying it into your script or are you running the sample that I provided. 

jacob_loughridge
Emerging Contributor

I hadn't applied any of it, I was just asking what that line was doing. Looks creating something in a similar vein to what you have here, will greatly improve performance and actually work better. I appreciate the reply and suggestion.  I hadn't looked into that arcpy geometry before, but it should be very useful for creating a script that isn't just a bunch of simple selects and paste values like I already have. 

Thanks again, will be a real help!

0 Kudos
RPGIS
by
Frequent Contributor

Yes. It basically utilizes the geometry of the feature and runs an analysis using the arcpy geometry methods. I have made recommendations to people to avoid using selections in a script and simply utilize the capabilities of the cursors. The cursors themselves will accomplish any gis related task.

0 Kudos
jacob_loughridge
Emerging Contributor

I have tried to implement your script and am running into an issue. During the CheckProximity step, I am getting an AttributeError: 'NoneType' object has no attribute 'distanceTo'. 

I am guessing the GetRecords portion is not returning a value to iterate through this portion of the script. Please let me know if I am on the wrogn track here. 

 

Traceback (most recent call last):
File "CommunityHelpScript.py", line 41, in <module>
CheckProximity( PoleRecords , StreamRecords , 100 )
File "CommunityHelpScript.py", line 26, in CheckProximity
if Point.distanceTo( Line ) <= SetDistance:
AttributeError: 'NoneType' object has no attribute 'distanceTo'

Failed script Script...
Failed to execute (Script).

0 Kudos
RPGIS
by
Frequent Contributor

It could be it is having trouble reading the geometry of the line feature; depending on if you are using shapefiles vs featureclasses. It is generally recommended to avoid using shapefiles and use featureclasses in filegeodatabases since shapefiles are notorious for getting corrupted.

Another option is to read through all of the segments and all of the points in the segments to find the closest point.

Read through the line feature class to see if any shape value return. If there is a corrupted record, it will return as a null value.

0 Kudos