Truncate/Append across a distributed collaboration

Question

Hi everyone,

I am currently working with two hosted feature layers within an ArcGIS Notebook on AGOL:

Layer_Edits (from Portal, shared via Distributed Collaboration to AGOL)

Layer_Prod (in AGOL)

I am trying to find the best way to write a script where the Layer_Edits layer is appended/added to the Layer_Prod. I have a situation where either changes might be made to existing data or new data is added in the Layer_Edits.

Sync is enabled as a product of the Edits layer being a part of distributed collaboration. However, it's my understanding that the truncate/append function isn't supported if syncing is enabled. What are my options?

jcarlson · Answer

Gotcha. Well, being a smaller layer, truncate and append is still a good approach. For the dual log-in, it's pretty simple:

portal = GIS('your portal url')
agol = GIS('AGOL url')

source_fl = arcgis.features.FeatureLayer('some url', portal)
dest_fl = arcgis.features.FeatureLayer('some other url', agol)

As far as creds, there's no need to store your credentials in an unsafe manner. There's an entire article on authentication strategies, and I'm sure there will be one that fits your needs.

https://developers.arcgis.com/python/guide/working-with-different-authentication-schemes/

Rberning · Answer

Thank you Josh! We originally were intending to use the distributed collaboration to avoid storing credentials (and honestly, I wasn't aware that you could log into both at the same time within AGOL) and to have a bit of a time delay. I'm still fairly new to learning python and using it to automate tasks.

Originally, the idea was that if an edit happens on the feature layer in Portal, there would be a 24 hour built-in delay before it shows up in Production (that way there is time to catch an error if one's been made). This is a relatively small dataset - around 100 features or less. However, we have about 80+ view layers that would be sourced off of the production layer.

jcarlson · Answer

You don't even need to use a distributed collab, just connect to both portals.

For truncate / append, you can just use FeatureLayer.delete_features(where='1=1') to delete everything in a layer. It won't reset the objectid to 0 like truncate does, but it works well enough when Sync is turned on.

Then you can either use FeatureLayer.append() or FeatureLayer.edit_features(adds=some_featureset) to load in the new data.

Another thing to consider: depending on how many features get edited in the Layer_Edits source, you can use GeoAccessor.compare() to compare the source and destination to identify which rows were added, updated, or deleted between the two, You can take the results of the comparison to make any changes needed to the destination, leaving unchanged rows in place.

For instance, we have a parcel layer with ~60,000 features in it. On any given day, only a few hundred are edited, the rest are unchanged. We use a comparison process to isolate the rows which have changed, then pass those adds, updates, and deletes to the service without truncating anything at all. Our hosted layer is brought up to date with the source in mere seconds, and the layer is never emptied of its data, so we can run those sorts of updates at any time of day or night without interrupting our users.