Bulk delete 570k features

[AFackler_NAPSG]

Hello all,

I need to delete over 570,000 waypoint features from a feature layer and have been running into trouble doing so. I am not looking to clear out the entire database, only waypoints that I have marked for deletion. Those marked use an existing field called "delete_this" and are marked as "yes_delete". However whenever I follow the steps from this document https://support.esri.com/en-us/knowledge-base/how-to-batch-delete-features-from-an-arcgis-online-hos... it always times out. Is there any other route I can take? Thanks!

[George_Thompson]

Do you have ArcGIS Pro?

If so, you can bring in the hosted layer and then open the table and delete. That should be able to handle that volume. I may take some time.

[AustinAverill]

Have you done this with any success recently? Probably 8 or so months ago I could not delete more than 2500 features at a time through Pro without getting a timeout error.

It would delete features, but after it got around that 2,500 mark in its loop it would just error out.

[AustinAverill]

I usually handle this with Python in batches of 2,000.

The code would look something like this:

from arcgis.gis import GIS
import pandas as pd

#use your own credentials here
org_url = 'https://myorg.maps.arcgis.com'
_un = 'myusername'
_pw = 'mypassword'

#get id of feature service
feature_service_id = '12345612345fe'

#enter layer index on service
layer_index = 0

gis = GIS(org_url, _un, _pw)

fl = gis.content.get(feature_service_id)[layer_index]

#returns all features marked delete as a spatial data frame
features_to_delete = fl.query(where="delete_this = 'yes_delete'").sdf

#loop through marked deleter features and call delete in batches of 2000 to avoid time out
max_index = len(features_to_delete) - 1
prev_index = 0

while (prev_index + 2000) < max_index:
    next_index = prev_index + 2000
    subset = features_to_delete.iloc[prev_index : next_index]
    oids = list(subset['OBJECTID'])
    fl.edit_features(deletes=oids)
    prev_index += 20000
    
#deletes the final number of features when less than 2,000 left
subset = features_to_delete.iloc[prev_index : max_index]
oids = list(subset['OBJECTID'])
fl.edit_features(deletes=oids)
    

[George_Thompson]

I have not done that # of delete's on a service in a while......... the python way is a good way to do it in chunks.

I know that Pro should do this behind the scenes in the map / table.

[AFackler_NAPSG]

Thanks for your script! However when I I have tried to run it on my end, I am getting this error. Any suggestions?

[AustinAverill]

On line 17 in my code, I made an error. It should read:

fl = gis.content.get(feature_service_id).layers[layer_index]

Sorry about that. Typed it out in a hurry yesterday. Lol

[JonathanMcD]

Deleting 570k points, how many will be left?

[RTPL_AU]

@AFackler_NAPSG 
You've made a backup?  That should be the first step.
For large datasets I'd use a python script in Pro to get the FGDB down to a local spot. Exporting from within AGOL can take a long time. 

Once you have a backup you can try deleting in Pro using the script above (concur, manual deletes will take forever).

You also have the option to create a Notebook in AGOL and run processes from there. I've not yet needed to do bulk deletes from there but should be feasible and removes Pro and your internet connection from the equation.

As @JonathanMcD  alluded to - how many points should remain?
Is it feasible to make a local backup, extract what you want to keep, and then overwrite the service with the extract? 

Editor tracking and attachments can make it interesting and are to be considered.

[AustinAverill]

I probably wouldn't advise doing a bulk delete through AGOL notebooks. At least regularly. Even with the above script being as efficient as the API will allow, deleting 570k features is going to take time, and in turn credits from an AGOL Notebook.

If you're Credit rich, go for it. Otherwise probably best to do it through a Pro Notebook or create a local installation of python to handle the task.