topic Re: arcpy.da.SearchCursor skipping rows in Python Questions

arcpy.da.SearchCursor skipping rows

jacob_loughridge — Fri, 13 Sep 2024 15:23:50 GMT

I have a script that is using the OID@ as the cursor to iterate through a feature class. It seems to be skipping 3 OIDs at random intervals during the script when using UpdateCursor, but seems to have a consistent 3 row skip when using SearchCursor.

Is there a reason as to why I am getting a random skip here?

with arcpy.da.SearchCursor(IterateLayer, ["OID@"]) as cursor: for row in cursor: rowID = str(row[0]) arcpy.AddMessage(fr"Processing for Pole: "+ rowID) PoleSelected = mgt.SelectLayerByAttribute(IterateLayer, "NEW_SELECTION", "OBJECTID = {}".format(row[0])) arcpy.AddMessage("Closest River/Stream Processing") def Closest_RiverStream(SaR, PoleS): def RiverStream_ID(PoleS): with arcpy.da.SearchCursor((PoleS), "Stream_ID") as cursor: for row in cursor: StreamID = '{}'.format(row[0]) StreamID = str(StreamID) return StreamID RID = RiverStream_ID(PoleS) RID = str(RID) clause = "OBJECTID = " + RID StreamSelected = mgt.SelectLayerByAttribute(SaR, "NEW_SELECTION", where_clause=clause) def StreamPermanence(S_S): with arcpy.da.SearchCursor(S_S, "StreamPermanence"): for row in cursor: StreamPerm = '{}'.format(row[0]) StreamPerm = str(StreamPerm) return StreamPerm if (RID != "-1"): ParclOwner = mgt.CalculateField(PoleS, "StreamPermanence", "'"+str(StreamPermanence(StreamSelected))+"'") else: NotInSearchDist = "Not Within 100 Feet of Stream or River" ParclOwner = mgt.CalculateField(PoleS, "StreamPermanence", "'"+str(NotInSearchDist)+"'") def StreamSize(S_S): with arcpy.da.SearchCursor(S_S, "FPAStreamSize"): for row in cursor: Stream_Size = '{}'.format(row[0]) Stream_Size = str(Stream_Size) return Stream_Size if (RID != "-1"): ParclOwner = mgt.CalculateField(PoleS, "StreamSize", "'"+str(StreamSize(StreamSelected))+"'") else: NotInSearchDist = "Not Within 100 Feet of Stream or River" ParclOwner = mgt.CalculateField(PoleS, "StreamSize", "'"+str(NotInSearchDist)+"'") #mgt.Delete(TempPole) Closest_RiverStream(StreamAndRivers, PoleSelected) mgt.SelectLayerByAttribute(StreamAndRivers, "CLEAR_SELECTION")

Re: arcpy.da.SearchCursor skipping rows

VinceAngelo — Fri, 13 Sep 2024 14:45:40 GMT

Never ever change the selection environment on a source while inside a cursor using that source.

Please use code formatting on your post so that the code indentation is legible.

- V

Re: arcpy.da.SearchCursor skipping rows

RPGIS — Fri, 13 Sep 2024 15:42:42 GMT

So, there are a couple of things to mention, not so much the question but issues that I see with the script itself.

Avoid nesting loops. In terms of performance, nested loops can be processing killers if they are used within themselves.
All cursor functions have SQL clauses that can be used, so having separate select by attribute query layer is unnecessary.
In my personal opinion, never set functions within the main portion of a script. I typically set them above the main processes and below the imports.

I would also recommend checking out arcpy geometry to get a better understanding of how to utilize feature geometries.

import arcpy from arcpy import ListFields from arcpy.da import SearchCursor as Searching , UpdateCursor as Updating # Retrieves records of an input layer/feature and returns a dictionary of values def GetRecords( Layer , ["OID@"] + Fields + ['SHAPE@'] , SQLClause ): SearchLayer = Searching( Layer , Fields ) if SQLClause is not None: SearchLayer = Searching( Layer , Fields , SQLClause ) return { row[0] : row[ 1:] for row in SearchLayer } # Returns a list of field names def GetFieldNames( Layer , ExcludeFields ): fieldnames = [ field.name for field in ListFields( Layer ) ] if type( ExcludeFields ) is list: fieldnames = list( set( fieldnames ).difference( set( ExcludeFields ) ) ) return fieldnames # Checks the distance between two geometries def CheckProximity( InputFeatureRecords , DistanceFeatureRecords , SetDistance ): WithinSetLimits = { } for IF_id , IF_Attributes in InputFeatureRecords.items(): for DF_id , DF_Attributes in DistanceFeatureRecords.items(): Point = IF_Attributes[ 0 ] Line = DF_Attributes[ 0 ] if Point.distanceTo( Line ) <= SetDistance: WithinSetLimits[ IF_id ] = DF_Attributes[ 0 ] else: WithinSetLimits[ IF_id ] = "Not Within 100 Feet of Stream or River" return WithinSetLimits Poles = '<PoleLayer>' Streams = '<StreamLayer>' PoleFields = GetFieldNames( infeature , '<[ list of field names to exclude ]>' ) StreamFields = [ "Stream_ID" , "StreamPermanence" ] PoleRecords = GetRecords( infeature , ["OID@"] + Fields + ['SHAPE@'] ) StreamRecords = GetRecords( infeature , ["OID@"] + Fields + ['SHAPE@'] ) CheckProximity( PoleRecords , StreamRecords , 100 )

The sample above isn't a solution but I just thought to provide something useful to give you a rough idea.

Re: arcpy.da.SearchCursor skipping rows

jacob_loughridge — Fri, 13 Sep 2024 15:44:32 GMT

I am not sure I follow, are you referencing the inital select by Attribute?

I have that part in a similar script and it doesn't skip around like this function. I added this function to the larger script and that is when the skipping began, as soon as I removed this function, the skipping stopped in the other script.

Re: arcpy.da.SearchCursor skipping rows

jacob_loughridge — Fri, 13 Sep 2024 16:08:46 GMT

Thanks, for the reply.

this portion is replacing the calculate field i have running in my script now correct?

            if Point.distanceTo( Line ) <= SetDistance: WithinSetLimits[ IF_id ] = DF_Attributes[ 0 ]
            else: WithinSetLimits[ IF_id ] = "Not Within 100 Feet of Stream or River"

Re: arcpy.da.SearchCursor skipping rows

RPGIS — Fri, 13 Sep 2024 22:41:15 GMT

I rewrote your script as such because it was really difficult to read and identify where the issue was.

Have you checked out the link in my previous post? It shows you how to utilize the geometry of a feature.

The line of code you are trying to use, are you merely copy and paying it into your script or are you running the sample that I provided.

Re: arcpy.da.SearchCursor skipping rows

jacob_loughridge — Mon, 16 Sep 2024 12:57:39 GMT

I hadn't applied any of it, I was just asking what that line was doing. Looks creating something in a similar vein to what you have here, will greatly improve performance and actually work better. I appreciate the reply and suggestion. I hadn't looked into that arcpy geometry before, but it should be very useful for creating a script that isn't just a bunch of simple selects and paste values like I already have.

Thanks again, will be a real help!

Re: arcpy.da.SearchCursor skipping rows

RPGIS — Mon, 16 Sep 2024 20:05:49 GMT

Yes. It basically utilizes the geometry of the feature and runs an analysis using the arcpy geometry methods. I have made recommendations to people to avoid using selections in a script and simply utilize the capabilities of the cursors. The cursors themselves will accomplish any gis related task.

Re: arcpy.da.SearchCursor skipping rows

jacob_loughridge — Mon, 16 Sep 2024 20:10:14 GMT

I have tried to implement your script and am running into an issue. During the CheckProximity step, I am getting an AttributeError: 'NoneType' object has no attribute 'distanceTo'.

I am guessing the GetRecords portion is not returning a value to iterate through this portion of the script. Please let me know if I am on the wrogn track here.

Traceback (most recent call last):
File "CommunityHelpScript.py", line 41, in <module>
CheckProximity( PoleRecords , StreamRecords , 100 )
File "CommunityHelpScript.py", line 26, in CheckProximity
if Point.distanceTo( Line ) <= SetDistance:
AttributeError: 'NoneType' object has no attribute 'distanceTo'

Failed script Script...
Failed to execute (Script).

Re: arcpy.da.SearchCursor skipping rows

RPGIS — Tue, 17 Sep 2024 18:07:49 GMT

It could be it is having trouble reading the geometry of the line feature; depending on if you are using shapefiles vs featureclasses. It is generally recommended to avoid using shapefiles and use featureclasses in filegeodatabases since shapefiles are notorious for getting corrupted.

Another option is to read through all of the segments and all of the points in the segments to find the closest point.

Read through the line feature class to see if any shape value return. If there is a corrupted record, it will return as a null value.

Re: arcpy.da.SearchCursor skipping rows

jacob_loughridge — Wed, 18 Sep 2024 18:56:45 GMT

I think issue was mainly due to the features being in different projections (comparing a local dataset to a national dataset). Instead of using distanceTo, I switched over to running the analysis Near geoprocessing tool, and comparing the if any of the NEAR_FID values I get match the Keys from the Line Dataset. After it is just a simple translation of data between the tables and utilize update Cursor to update the feature class.

Thanks for the help, you pointed me in the right direction and helped me better understand how to utilize geometry and the best practices for search/update cursor.

def GetRecordsDict(IterateLayer, Fields): rows = {} with arcpy.da.SearchCursor(IterateLayer, Fields) as cursor: for row in cursor: key = row[0] values = list(row[1:]) rows[key] = values return rows def Proximity(PoleFeature, DistanceFeature, Distance): for keyA, value in PoleFeature.items(): near_fid = value[0] if near_fid in DistanceFeature: value = list(value) value[1] = DistanceFeature[near_fid][0] value[2] = DistanceFeature[near_fid][1] PoleFeature[keyA] = value else: value = list(value) value[1] = "Not Within 100ft of Stream" value[2] = "Not Within 100ft of Stream" PoleFeature[keyA] = value arcpy.AddMessage("Updated Dictionary for Poles") return PoleFeature def UpdateFeatureClass(UpdateDict, FC, Fields): with arcpy.da.UpdateCursor(FC, Fields) as cursor: for row in cursor: key = row[0] if key in UpdateDict: row[1:] = UpdateDict[key] cursor.updateRow(row) arcpy.AddMessage("FC updated Properly")

Re: arcpy.da.SearchCursor skipping rows

RPGIS — Thu, 19 Sep 2024 13:02:17 GMT

Hi @jacob_loughridge

Just to give you another tip about coding, especially in GIS, it is best to reduce using gp tools in a script or mid script since those create other features that then need to be utilized, which further complicates scripts.

The one cool thing about the geometry methods in arcpy is you can also project features to different projections using projectAs (spatial_reference, {transformation_name}) geometry method.