Updating Sync-Enabled Feature Layers Via Python Code

[ChristopherPouliot]

My organization would like to publish copies of our enterprise data to our Portal so that it can be used in Field Maps as reference layers in disconnected situations.  Data such as our Parcel Layer would be very helpful to our field workers.  This requires that the Feature Layers be set as Sync-Enabled.  Since our enterprise data updates periodically we would need a way to keep those reference layers up to date.  From everything I've read there is currently not a way to update existing data in a sync-enabled layer -- See "Allow Offline Editing" section of this doc -- https://doc.arcgis.com/en/arcgis-online/manage-data/manage-editing-hfl.htm#ESRI_SECTION2_D884E7592F5...

Well, there is a way, but it involves turning Sync-Enabled off while updating which would break any WebMap and Checked Out Area that uses that layer.

We have 1000+ layers that we would like to host in Portal but I would need an automated way (Python) to keep them up to date.  Any ideas would be greatly appreciated!  Thanks.

[jcarlson]

Are you referring to this?

If synchronization is enabled, you can append new features to the hosted feature layer but not update existing features using the Update Data option on the layer's details page.

Note that "Update Data" in this case is specific to one of the functions in the API; your ability to "update data" (no caps) is not impaired by whether or not Sync is enabled.

What you can do is pass updates to the hosted layer using FeatureLayer.edit_features(updates=[]). If you have edit timestamps on your data, you can selectively pass in only those features that have been edited within a given timeframe, too.

At Kendall County, we have about 50k parcels, but only a couple hundred are edited any given day, so we use the Python API together with the pandas module to identify only those rows that have been edited, and then we edit those features in place.

Let me know if this sounds useful to you, I can share a generalized version of the code we use for it. We have about 30 hosted layers that are kept up to date with offline sources on a nightly basis, so I can tell you it's certainly an adaptable approach.

[KansasDASC]

Hi @jcarlson would you be able to share a generalized version of your code for this? We're also in the same boat as the OP and it would be helpful if we could routinely update a feature layer we're using as the basis of a FieldMaps app while it has sync enabled. I don't want to just append new features because there will be new features and updates to existing ones in the updates.

[ChristopherPouliot]

Thanks for the response, Josh.  Yes, that is what I was referring to.  I hadn't seen the edit_features function and that is definitely a step in the right direction.  The issue we have is that our data sources come from dozens of authoritative sources and we don't have control over their schemas -- such as adding last_edited dates or globalids to the data.  We know when a dataset has been updated but don't have a reliable way of determining which records were added/deleted/updated without doing a record by record comparison between source and destination.  That is why I was leaning heavy on ESRI's API to do the change detection for us which it can do - just not with Sync enabled.

A couple options I've thought of include:

• Write a change detection function which would, for performance sake, require downloading the layer down from Portal and do the comparisons with arcpy.  Then push the changes up with the edit_features function
• Push all our layers into an SDE database and use referenced feature layers (instead of hosted).  Then the update process would all be local with a less black-boxy database.  Storing 1000 data resources in SDE causes its own set of headaches for us though.

I'd be interested to know if ESRI is planning to provide the ability to expand their update process to include Sync-enabled layers.  That would help me to know how far I should travel down other rabbit holes for another solution.

[jcarlson · Accepted solution]

Ah, well. We don't have control over most of the schemas we work with, and generally, unless you're dealing with the actual shapes, it's not required.

That "record by record comparison" is very simple if you use pandas' compare function.

Also, once you read / query / load your source data, you can modify the schema as needed on the resulting DataFrame to ensure that it matches with the destination.

I actually just gave a talk about this yesterday at a conference. See if anything in here helps you out:

https://github.com/jdcarls2/ilgisa-2022/blob/main/hosted-copy/hosted-copy.ipynb

[ChristopherPouliot]

Thanks a ton Josh.  It's good to know others are in the same boat as we are.  You've given me some good ideas to run with.