topic Deleting Features: Does cursor mess with order? in Python Questions

Deleting Features: Does cursor mess with order?

MatthiasBuehler — Sun, 25 Sep 2016 09:47:22 GMT

Hi,

Question:

I have read that cursors can only be iterated forward, not in reversed order.

Not if I have e.g. 5 features and the following code structure:

delCursor = arcpy.da.UpdateCursor(fc, field)
for row in delCursor:
if testIfRowNeedsToBeDeleted(..):
delCursor.deleteRow() # messes with the number of elements in the loop!!

How can I guarantee that the correct features are deleted when I delete features with 'early' OBJECTIDs out of the list, as after the first deletion, the number of elements in the field is already not the same as the cursor's number.

How is this handled internally?

* * *

In the end, I just need to delete specific rows out of a feature class, depending on specific field values and I want to avoid any issues with the order.

My favourite would be this structure in pseudo code:

to deleteFeatureOBJECTIDs = []
loop over cursor:
if feature needs to be deleted based on multiple field values:
deleteFeatureIDs.append(current feature)
arcpy.deleteFeaturesInFC(deleteFeatureOBJECTIDs)

[[ I have also tried code-based creation of the SQL 'where_clause', with a string list, but this type of search crashes ArcGIS PRO quite easily. ]]

Any input welcome!

Thanks!

matt

Re: Deleting Features: Does cursor mess with order?

MatthiasBuehler — Sun, 25 Sep 2016 10:02:16 GMT

ps.

if possible, I'd like not to have to operated with temp layers.

Re: Deleting Features: Does cursor mess with order?

NeilAyres — Sun, 25 Sep 2016 11:01:52 GMT

Not quite sure what you are saying here about affecting the order.

If you run an update and then delete specific rows based on some criteria, the record will be deleted.

The row order, OID based is fairly arbitrary anyway, based on the order of creation.

So, if you have OIDs 1, 2, 3, 4, 5, then you delete 2 & 3, you will be left with OIDs 1, 4 & 5.

It will not automatically re-order or re number the remaining OIDs.

To do that you would have to do a copy features or something to a new table.

Re: Deleting Features: Does cursor mess with order?

DanPatterson_Retired — Sun, 12 Dec 2021 00:44:50 GMT

If you have to use cursors, be positive thinking and keep those features that you want to keep and save to a new featureclass.

You can put this in cursor-esque

awl = [1, 2, 3, 4, 5]
>>> dump = [2, 4]
>>> keep = [i for i in awl if i not in dump]
>>> keep
[1, 3, 5]‍‍‍‍‍

Re: Deleting Features: Does cursor mess with order?

MatthiasBuehler — Sun, 25 Sep 2016 13:10:49 GMT

Hi guys, thanks for the replies - on a sunday!!

Oh - I think the key is in understanding that the OBJECTID in a FC is NOT always enumerated from 1 to n. I was under the impression this is the case. (I'm not a GIS expert)

I have just tried manually deleting a feature and adding an other one manually and the result shows me that the OBJECTID of the newly drawn does not replace the last (missing) spot, but is 1 higher than the highest existing OBJECTID value.

-->

1] Can you guys confirm that arcpy handles this the same as when doing stuff manually?

2] If so, then the whole question would be answered by this very important detail.

Thanks guys!

matt

Re: Deleting Features: Does cursor mess with order?

DanPatterson_Retired — Sun, 25 Sep 2016 17:31:51 GMT

why don't you run a comparison on your data, since you will then have a standard to compare to. Of course, it goes without saying, one never alters an existing dataset, but uses a copy. Storage is cheap, time is not.

Re: Deleting Features: Does cursor mess with order?

RebeccaStrauch__GISP — Sun, 25 Sep 2016 18:38:25 GMT

I would say yes, manual vs script will handle things the same, assuming all else is equal.

you may want to read thru Fundamentals of ObjectID fields—Help | ArcGIS for Desktop to get a little better understanding or OIDs.

Since we don't really have control over the OID assignment, I suggest adding a unique field to store the value of the order (if that's what you are going for). If important, you could start out by calculating the start value as the current OID value. Then, before you delet the on record, grab the value and insert it in the "uniqid" field for the new record.

but as Dan mentioned, always good to compare output from manual vs script (and on a copy). Always good to test and review, especially if this is critical and will be used "blindly" (leap of faith) in the future.

sorry not including any code...on my iPad.

Re: Deleting Features: Does cursor mess with order?

XanderBakker — Sun, 12 Dec 2021 00:44:53 GMT

Hi Matthias Buehler ,

When deleting elements from a collection you have to be careful not to "cut the branch you sitting on":

Source: http://www.publicdomainpictures.net/pictures/10000/velka/1210-1240955295Hn8Q.jpg

The cursor however, handles this correctly (as far as I know). Look at the snippet below:

def main():
    import arcpy

    fc = r'C:\GeoNet\DeleteFeaturesPython\data.gdb\points'
    print "before", getCount(fc)

    with arcpy.da.UpdateCursor(fc, ('OID@')) as curs:
        for row in curs:
            oid= row[0]
            if oid % 2 == 0:
                # delete even oid's
                curs.deleteRow()

    print "after", getCount(fc)


def getCount(fc):
    return int(arcpy.GetCount_management(fc).getOutput(0))


if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It prints in my case:

before 876
after 438‍‍

And looking at the attribute table before and after it has removed correctly the rows with en even ObjectID:

before after

Might be something specific with the case you are working on.

BTW: love your CityEngine work on Independence Day: Resurgence

Re: Deleting Features: Does cursor mess with order?

MatthiasBuehler — Mon, 26 Sep 2016 07:41:46 GMT

Hi guys,

Thanks for all the inputs, much appreciated.

And yes, the features I'm deleting are derived from some other features, so if the input features have changed, I have to delete the irrelevant derived features.

Cool pic, btw!

Xander Bakker‌

Thanks for the compliment! I'm good in some things, but not so in others. Well, like all of us.

Did you see this presentation?

Arcpy is a new thing I'm learning for my CityEngine services, so I'm trying to get the things I need in a workflow prototype I'm working on for Esri Germany to work.

Huge thanks!

Matt

Re: Deleting Features: Does cursor mess with order?

XanderBakker — Mon, 26 Sep 2016 11:27:48 GMT

Hi Matthias Buehler ,

If you can share a bit more information and perhaps some (dummy) data, maybe I can track what is happening and see if there is something in the Python script you're using.

And yes I attended the live-seminar where you presented your work for Independence day.

Kind regards, Xander

Re: Deleting Features: Does cursor mess with order?

JoshuaBixby — Mon, 26 Sep 2016 14:18:16 GMT

If you can consistently crash ArcGIS Pro by using a specific SQL WHERE clause, I suggest you contact Esri Support and get a bug logged. Afterwards, share the bug number and information back in this post for the benefit of the community.

Whether accessing data sets through the GUI, ArcPy cursors, or some other client/API; there are multiple layers of abstraction between the tool you are interacting with and the data itself. It is the lower levels in these layers/stacks that actually implement updates, insertions, and deletions; and this is where data integrity is implemented.

As others have commented already, "trust the cursors." From a purely practical perspective, if ArcPy cursors could so easily corrupt data sets, they wouldn't be around any longer. There are few quicker ways to lose customers than having tools that corrupt data easily and consistently. From a theoretical perspective, understanding why your first code snippet will work and not corrupt data involves learning about SQL, database engines, result sets, cursors, transactions, and other elements of databases. If you want to avoid taking a course or cracking a textbook, there is a not-so-primer primer on Understanding how SQL Server executes a query. Although the blog post is specific to SQL Server, the principles apply to databases in general.

Re: Deleting Features: Does cursor mess with order?

LukeWebb — Tue, 27 Sep 2016 15:35:00 GMT

Heres my code for making an SQL statement from a list with no error catching or anything:

I use it with lists of ObjectIDs all the time to start my cursors

fieldName = 'FIELD1'
valueList = ['Value1', 'Value2']

where = "%s in ('" % fieldName
where += "','".join(map(str, valueList)) + "')"
print where

FIELD1 in ('Value1','Value2')