topic Re: What is the recommended method to overwrite feature layer data using Python API? in ArcGIS API for Python Questions

What is the recommended method to overwrite feature layer data using Python API?

A_Schwab — Fri, 19 Jan 2024 10:15:16 GMT

I have a feature layer that I'm trying to update regularly with new data using the ArcGIS Online Python API.

The two methods I've found in ESRI docs are to create a featurelayer collection and use overwrite() or to use the Truncate and Append method.

The overwwrite() method doesn't change any of the data but says the operation is successful, so there's no indication why it hasn't worked.

The truncate and append method fails at calling append() with an unspecified 500 error.

I would love to know what the officially supported way to overwrite an existing layer on AGOL is, either from a spatially-enabled data frame or from a layer in a file geodatabase.

Re: What is the recommended method to overwrite feature layer data using Python API?

jcarlson — Fri, 19 Jan 2024 14:11:04 GMT

Truncate and append is the method to use. Overwriting a feature service has historically led to problems.

Don't use the literal append() method, either. If you've got a spatial dataframe, you can just do

FeatureLayer.edit_features(adds=spatial_dataframe.spatial.to_featureset())

Re: What is the recommended method to overwrite feature layer data using Python API?

A_Schwab — Fri, 19 Jan 2024 15:48:58 GMT

Looks like `edit_features()` does what I need so I'll go ahead with that. Thanks for the info!

Re: What is the recommended method to overwrite feature layer data using Python API?

LRoberts_RDM — Thu, 01 Aug 2024 23:37:43 GMT

Hey @jcarlson, I am currently losing my mind to this issue. I'm curious if you have a workflow for getting data into a spatial dataframe? Currently I am exporting data from enterprise gdb to a file gdb, zipping that up and pushing to AGOL, then using FeatureLayer.append() to update my hosted feature service. I'm having wildly inconsistent succes/fails with the append portion. Sometimes it works, and other times it returns "Unknown Error (Error Code: 500).

Will the spatial data frame work if I have 90,000 polygons I need to push into a hosted feature service?

Thank you!

Re: What is the recommended method to overwrite feature layer data using Python API?

jcarlson — Fri, 02 Aug 2024 12:26:07 GMT

I put a generic version of the process in this repo:

https://codeberg.org/kendall-county-gis/project-templates/src/branch/main/generic-etl

To pull your stuff from an EGDB, you might want to look at pandas.read_sql() in conjunction with arcgis.features.GeoAccessor.from_df(). I've never used it to query an EGDB directly, so maybe there's a method in the arcgis Python API to do so more directly?

Re: What is the recommended method to overwrite feature layer data using Python API?

LRoberts_RDM — Mon, 05 Aug 2024 22:01:46 GMT

I have it working from a file gdb at the very least. So far so good. Thank you for this insight. It's a life saver!

Re: What is the recommended method to overwrite feature layer data using Python API?

LRoberts_RDM — Tue, 13 Aug 2024 16:00:45 GMT

Just a quick follow up note regarding this thread. I ran into issues using the edit_features(adds=featureset) because my feature set was too large (approx 80,000 polygons). I was able to make the overwrite work using the following:

output_zip = [path to zip] gis = GIS("https://www.arcgis.com", username, password) item = gis.content.get([item id]) flc = FeatureLayerCollection.fromitem(item) flc.manager.overwrite(output_zip)

One very important note that I should add is that at first, I was using the shutil python library to zip my file geodatabase. I would get an error during the overwrite. I changed that to the zipfile library, and it worked like a champ.

Re: What is the recommended method to overwrite feature layer data using Python API?

jcarlson — Tue, 13 Aug 2024 16:05:13 GMT

With that many, you'll want to batch the updates.

i = 0 batchsize = 200 while i < len(df): batch = df.iloc[i:i+batchsize] fl.edit_features(adds=df.spatial.to_featureset()) i += batchsize