topic Re: How to update sync data on ArcGIS Online in ArcGIS Online Questions

How to update sync data on ArcGIS Online

JordanMiller4 — Sat, 23 Nov 2019 16:15:33 GMT

What's the best method for updating ArcGIS Online data that has sync enabled? Currently, we have multiple offices actively using this data offline.

Re: How to update sync data on ArcGIS Online

Anonymous User — Wed, 29 Jan 2020 00:10:35 GMT

Hi Jordan Miller‌,

With sync-enabled Hosted Feature Layers in ArcGIS Online, editors can sync offline edits at any time. Ideally this would be with a strong WiFi connection.

Manage hosted feature layers—ArcGIS Online Help | Documentation

Do you mean update the Hosted Feature Layer as in append new features or overwrite from a desktop client?

Thanks,

-Peter

Re: How to update sync data on ArcGIS Online

NickShannon2 — Fri, 07 Aug 2020 06:22:00 GMT

Yes, update a Hosted Feature Layer with sync enabled.

It is not possible to overwrite a Feature Hosted Layer with sync enabled:

https://community.esri.com/message/845787-re-cannot-overwrite-hfs-with-views?commentID=845787#comment-845787

I've investigated this script - but can't get it to work for me.

https://community.esri.com/docs/DOC-11621-update-hosted-feature-service-with-feature-class

Scenario: Maps made for Collector usually have reference layers, such as, road network. These layers need to go offline (sync enabled) and need to updated on occasions.

From what I've read, the only way to update layers with sync enabled is to append.

I'm currently looking at using the geoprocessing functions: delete rows & append to update these layer.

But, as Jordan Miller asks, it would be good to know what is Esri's recommended method.

Re: How to update sync data on ArcGIS Online

Anonymous User — Mon, 10 Aug 2020 20:24:28 GMT

Hi Nick Shannon‌ and Jordan Miller‌,

As far as the best path forward, I think you have two options.

These depend on the end goal of the data changes. Both of these operations can be scripted.

With a sync-enabled layer, as you mentioned you can add features via append. Updating features through append is not possible with sync enabled layers. Append would be the best option if you want to add thousands of features to the service. Deleting rows and appending to the hosted feature layer works as well. Append documentation: https://doc.arcgis.com/en/arcgis-online/manage-data/manage-hosted-feature-layers.htm#APPEND

If the data updates are smaller and contain adds, updates and deletes, using a scripted applyEdits request is the best option. I did some quick testing to confirm, and all three operations are allowed with offline replicas present. My offline replicas with edits synced without issue too. Here is our developer documentation on this: https://developers.arcgis.com/rest/services-reference/apply-edits-feature-service-.htm

I hope this helps,

-Peter

Re: How to update sync data on ArcGIS Online

Anonymous User — Fri, 11 Sep 2020 19:51:12 GMT

Hi Peter Klingman‌,
From our enterprise gdb, we've published a hosted feature layer on AGO, with sync enabled. The purpose of the AGO layer is 1) for use in a variety of web maps and applications and 2) to take offline as a reference layer in Collector or Field Maps.

The feature class in our enterprise gdb contains about 700 features, and undergoes a small number of edits on a weekly basis including adding features, editing attributes or occasionally deleting a feature. I need to be able to update the AGO layer to reflect changes. I cannot use overwrite because sync is enabled (and users have a few offline areas downloaded to their devices).

I'd like to have a script push the on premise edit to the AGO FL, and schedule it to run nightly. The workflow I'm describing seem similar to Nick Shannon's use case. For my scenario, is it your recommendation that we use Py API to identify the set of features that have been edited via add, update or delete and then use Apply Edits to push the edits to AGO? Would you have an example script to share?

Or is there a better way? We have several key datasets for which I'll need to implement a best practice solution. Thanks.

Re: How to update sync data on ArcGIS Online

Anonymous User — Wed, 16 Sep 2020 01:27:37 GMT

Hi Katie Andrews‌,

With a small number of edits, I think a script that uses the ArcGIS API for Python's Edit_features function (example here) could be a good choice. I think what could be challenging here is linking the changes made in the Enterprise GDB to the features in the Hosted Feature Layer that need to be updated/deleted.

If you have ArcGIS Enterprise, Distributed Collaboration would be good to look into for this scenario.

Per my recommendation above and the direction Nick was investigating, you may also want to look into using the Delete Features—ArcGIS REST API | ArcGIS for Developers Rest API function to delete all your features (essentially truncating your layer) and then Append (Developer reference: Append (Feature Service)—ArcGIS REST API | ArcGIS for Developers) a FGDB copy of the up-to-date Enterprise GDB dataset to the Feature Layer. This would be advantageous because you wouldn't need to detect changes in the Enterprise Geodatabase, and this workflow works with Sync-enabled layers and offline replicas. Something to keep in mind is that with this workflow, when the FGDB is appended to the empty layer it creates new records. You would want to make sure that any workflows using the feature layer don't depend on Object ID or Global ID. As you mentioned your feature layer is used as only a reference layer offline, so while syncing back features that have since been truncated/appended would create a new feature (possibly resulting in a duplicate), this does not appear to be consequential.

I will say that best practices and examples for updating data in ArcGIS Online is something that we're looking to put out a series of detailed blogs on. The variety of different scenarios (e.g. update frequency, source data, IT setup, workflows the feature layer is used in, etc.) that require a different data update approach has made a recommendation of a single best practice challenging, but we are hoping that the blog series will help fill this gap a little bit. It will contain info on applyEdits, append, distributed collaboration, and a few other methods as well.

Thanks,

-Peter

Re: How to update sync data on ArcGIS Online

NickShannon2 — Wed, 16 Sep 2020 01:51:16 GMT

I will say that best practices and examples for updating data in ArcGIS Online is something that we're looking to put out a series of detailed blogs on.

This would be useful!

I've created a simple model for 'delete rows' & append new data. I've tested it out & it works fine. New features pop-up on the Collector app in offline mode (once synchronised). Will be testing it out for the cadastra dataset — which is quite large.

I was hoping ArcGIS Pro would have an out-of-the-box update methodology for sync-enabled layers. But I appreciated there are many varied scenarios for updating data — as you mentioned.

Re: How to update sync data on ArcGIS Online

Anonymous User — Tue, 22 Sep 2020 22:54:52 GMT

Thank you Peter Klingman

Re: How to update sync data on ArcGIS Online

Anonymous User — Mon, 14 Dec 2020 21:18:46 GMT

Hi @Anonymous User,

What's considered a 'small number of edits'? A blog series on this and the pros and cons of Collaboration would be very useful. I've heard that Collaboration doesn't work well if data is updated via delete rows/truncate and append in the Enterprise Geodatabase.

Re: How to update sync data on ArcGIS Online

Anonymous User — Mon, 14 Dec 2020 23:05:05 GMT

Hi Mark,

It is not recommended to commit more than 2,000 edits per request using applyEdits.

You are correct that distributed collaboration would be less performant when truncating and appending in the Enterprise GDB. The reason for this is that if you truncate/append in the Enterprise GDB, when the replica is synced to Online it will be the entire dataset. Without truncating/appending in Enterprise, only the first sync would be the entire dataset and subsequent ones are just deltas, meaning fewer records and faster sync times.

I hope this helps, and I appreciate the blog suggestion.

Thanks,

-Peter

Re: How to update sync data on ArcGIS Online

Mondi_GIS — Fri, 29 Jan 2021 09:32:47 GMT

I have a roads dataset with 90000+ features and a polygon dataset with 20000 features - is delete and append still the only/best way to update this when Sync is enabled and users have it in use in collector?

Re: How to update sync data on ArcGIS Online

PhilLarkin — Fri, 29 Jan 2021 19:20:45 GMT

@Anonymous User Do you have an update on the blog posts you mentioned?

Re: How to update sync data on ArcGIS Online

Anonymous User — Mon, 01 Feb 2021 19:27:25 GMT

Hi @PhilLarkin they're still in the works, sorry for the delay. We do hope to release them as soon as we can.

Re: How to update sync data on ArcGIS Online

Mondi_GIS — Mon, 29 Mar 2021 14:33:24 GMT

We are still struggling with this. The delete and Append route crashes multiple times and we have to keep rerunning the processes until they have delete and appended all the data. It becomes an extremely manual process and hugely time consuming.

Re: How to update sync data on ArcGIS Online

HeatherWidlund — Tue, 06 Apr 2021 16:52:12 GMT

I am struggling with this, too. I have an open support case with no resolution after weeks and weeks (includes the non-updating of Field Maps data when the data is updated using this method). I started doing the delete and append in Notebook on AGOL to try to avoid connection errors, but it still fails sometimes. I was hoping to automate it with Notebook when that functionality comes online next week but if it's still going to fail half the time that won't work either.

Re: How to update sync data on ArcGIS Online

NickShannon2 — Mon, 04 Oct 2021 08:08:49 GMT

Just revisiting this.

As mentioned by @Mondi_GIS & @HeatherWidlund — the 'delete rows' Python API tool struggles with large datasets. @HenryLindemann provided a great script to help with this: Re: Delete features and error 504 and timed out - Esri Community (even this script stopped a few times — my dataset has over 2 million records!).

But, even if you get the script to work consistantly, there's another problem: data replicas. My FHS grew to over 14GB due to replicas. You can delete the replicas , but this is a problem if users have taken the dataset offline in FieldMaps etc.

So, I've gone back to the drawing board — I think the way to go is to provide users with an MMPK or a VTPK package of the data. Host the package in ArcGIS Online, and update as necessary. The package can be linked to a map, so it gets downloaded in FieldMaps — easy for the user. Certainly much easier than updating a large sync enabled dataset.

Prepare maps for download—ArcGIS Field Maps | Documentation

I'll test this out and and will report back.

Re: How to update sync data on ArcGIS Online

MatthewStephenson2 — Thu, 27 Jan 2022 23:26:07 GMT

@NickShannon2, have you tested your package idea yet? I am dealing with a similar scenario and looking for options that will work for my organization. I'm interested to know how this idea worked out for you in practice.

Re: How to update sync data on ArcGIS Online

Anonymous User — Fri, 28 Jan 2022 01:41:16 GMT

I'd probably look at an alternative route as our MMPK's or VTPK's would be rather large for our service areas. It's just not feasible for them to be updated regularly by users.

Something to consider:
https://community.esri.com/t5/arcgis-data-interoperability-blog/building-a-data-driven-organization-part-4/ba-p/1086265
https://community.esri.com/t5/arcgis-data-interoperability-blog/building-a-data-driven-organization-part-2-go/ba-p/1081060
At least one of the methods involves uploading a FGDB to ArcGIS Online and then appending from there to the hosted feature layer. This is far more performant as it's being done within the ArcGIS Online hosted data centre(s). The inbuilt Esri tools really don't seem to cope with large dataset changes.

Re: How to update sync data on ArcGIS Online

OliverIshmael — Tue, 14 Jun 2022 13:28:33 GMT

@Peter_Klingman Do you have an update on the blog posts you mentioned?

Re: How to update sync data on ArcGIS Online

Gavin_ZWORLD — Wed, 03 Aug 2022 00:18:12 GMT

Additional question about updating a large Sync enabled dataset.

In my case the data are used as a view-only reference in offline maps, would simply disabled Sync on the service, running a truncate and append, then reenabled Sync be a workable solution or would it case downstream issues?