What is the recommended method to overwrite feature layer data using Python API?

[A_Schwab]

I have a feature layer that I'm trying to update regularly with new data using the ArcGIS Online Python API.

The two methods I've found in ESRI docs are to create a featurelayer collection and use overwrite() or to use the Truncate and Append method.

The overwwrite() method doesn't change any of the data but says the operation is successful, so there's no indication why it hasn't worked.

The truncate and append method fails at calling append() with an unspecified 500 error.

I would love to know what the officially supported way to overwrite an existing layer on AGOL is, either from a spatially-enabled data frame or from a layer in a file geodatabase.

[jcarlson · Accepted solution]

Truncate and append is the method to use. Overwriting a feature service has historically led to problems.

Don't use the literal append() method, either. If you've got a spatial dataframe, you can just do

FeatureLayer.edit_features(adds=spatial_dataframe.spatial.to_featureset())

[LRoberts_RDM]

Hey @jcarlson, I am currently losing my mind to this issue. I'm curious if you have a workflow for getting data into a spatial dataframe? Currently I am exporting data from enterprise gdb to a file gdb, zipping that up and pushing to AGOL, then using FeatureLayer.append() to update my hosted feature service. I'm having wildly inconsistent succes/fails with the append portion. Sometimes it works, and other times it returns "Unknown Error (Error Code: 500).

Will the spatial data frame work if I have 90,000 polygons I need to push into a hosted feature service?

Thank you!

[jcarlson]

I put a generic version of the process in this repo:

https://codeberg.org/kendall-county-gis/project-templates/src/branch/main/generic-etl

To pull your stuff from an EGDB, you might want to look at pandas.read_sql() in conjunction with arcgis.features.GeoAccessor.from_df(). I've never used it to query an EGDB directly, so maybe there's a method in the arcgis Python API to do so more directly?

[LRoberts_RDM]

I have it working from a file gdb at the very least. So far so good. Thank you for this insight. It's a life saver!

[LRoberts_RDM]

Just a quick follow up note regarding this thread. I ran into issues using the edit_features(adds=featureset) because my feature set was too large (approx 80,000 polygons). I was able to make the overwrite work using the following:

output_zip = [path to zip]
gis = GIS("https://www.arcgis.com", username, password)
item = gis.content.get([item id])
flc = FeatureLayerCollection.fromitem(item)

flc.manager.overwrite(output_zip)

One very important note that I should add is that at first, I was using the shutil python library to zip my file geodatabase. I would get an error during the overwrite. I changed that to the zipfile library, and it worked like a champ.

[jcarlson]

With that many, you'll want to batch the updates.

i = 0
batchsize = 200

while i < len(df):
    batch = df.iloc[i:i+batchsize]
    fl.edit_features(adds=df.spatial.to_featureset())
    i += batchsize

[A_Schwab]

Looks like `edit_features()` does what I need so I'll go ahead with that. Thanks for the info!