Replication and Sync'ing Hosted Feature Layers to local gdb?

[deleted-user-t3dSviijg-m9]

Situation: We are using editor tracking with our Windows Domain Active Directory. It works just fine for edits made by users on ArcGIS Desktop. Problem... when field users use Collector with feature services coming from our ArcGIS for Server, Collector does not pass credentials for editor tracking, we get "ArcGIS_Anonymous" or whatever it is. I understand this is an inherent issue. So what are the workarounds? Hosting the features layers on ESRI's cloud? That's fine, it will keep track of editors, albeit, with their AGOL usernames and not our Active Directory, but will still give an indication of who edited it. 

The question: Is there a way to host those layers on ESRI's cloud, while maintaining a replicated copy of it? It seems like a pain to have to continuously download the data to get a local copy which becomes out of date quickly and requires another download, and then another, etc. etc. etc.

It just seems easier to host the data in the cloud, and have a replicated version sitting locally so data and be reconciled to ESRI's cloud and vice versa. 

The whole point is to be able to maintain editor tracking from field users, and try to avoid having to download copies from AGOL all the time. 

Would ArcGIS Online - collector app pass ArcGIS for Server authentication? 

[ScottPrindle · Accepted solution]

Hello Nick,

First we can cover the potential for replicating a hosted feature service to an enterprise geodatabase:

While it could be possible to replicate a hosted feature service to your enterprise geodatabase, there is not currently a supported/documented workflow. It may be possible through a script and use of the sync workflow. The createReplica operation can be used to generate JSON of the data, which could then be dumped into the enterprise geodatabase. The replica will persist on the hosted feature service, so edits on both ends can continue to be synced. This would be quite an undertaking in Python, but it is definitely something to consider if you must go that way. If you're interested in going down this rabbit hole we can definitely talk about this in more detail.

The second topic would be the recording of "Esri_Anonymous" as the identity for editor tracking:

"Esri_Anonymous" is the identity that is used when a service is unsecured. If the service is secured, then users will be prompted for credentials when accessing the service, and the credentials provided (in your case Active Directory) will be used as the identity.

Another caveat about this workflow is Collector and it's use of the identity. The authentication tier for your ArcGIS Server should be GIS-tier (as opposed to web-tier) so Collector can interact properly with your ArcGIS Server and pass the identity of field workers. GIS-tier authentication is token based, and Active Directory can still be used as the user store and role store. With GIS-tier, ArcGIS Server will check the credentials and issue a token for valid users. Web-tier authentication is handled by the web server (IIS or Java), so it acts as more of a gatekeeper. This configuration may be ideal for Single Sign-On workflows in web browsers and other clients, but isn't the best option when looking at editor tracking in Collector.

When you configure GIS-tier authentication and secure these services for use through Collector, you will see two credential prompts in the app. The first is to ArcGIS Online, the user needs to verify they are a part of your organization so they can view the web maps you've set up. The second prompt will be when they open the web map. This will verify they are allowed to view the secure feature service, but most importantly, they are providing that identity to be recorded for editor tracking. If they are using these maps offline, they should only have to provide the ArcGIS Server credentials when they first download the map and when they sync, because those will be the only times they are interacting with the service (and needing a token to prove they are who they say they are).

If any of this information is unclear, let me know! There's a lot of moving parts when it comes to Web GIS and I'd like to make sure everyone that comes across this issue is able to get what they need out of this conversation.

Thanks,

Scott

[ViTran]

Scott Prindle wrote:

While it could be possible to replicate a hosted feature service to your enterprise geodatabase, there is not currently a supported/documented workflow. It may be possible through a script and use of the sync workflow. The createReplica operation can be used to generate JSON of the data, which could then be dumped into the enterprise geodatabase. The replica will persist on the hosted feature service, so edits on both ends can continue to be synced. This would be quite an undertaking in Python, but it is definitely something to consider if you must go that way. If you're interested in going down this rabbit hole we can definitely talk about this in more detail.

Hi Scott,

I know the original question is quite old (> 2 years), but I'm now into the same problem that the OP had. I have a hosted feature service on AGOL that routinely needs to be synced back to the enterprise geodatabase. Please note that our organization's AGOL and enterprise GDB are separate. I'm interested in the solution that you wrote above. I wonder if you could post some more details about that solution.

Thanks!

[ScottPrindle]

Hello Vi Tran,

Unfortunately I don't have much of a solution for that. It was more of a conceptual discussion. The good news though is that the ArcGIS Platform has advanced a bit in the past two years with regards to Distributed Collaboration. This became available sometime after this discussion thread.

If Distributed Collaboration doesn't fit your needs, then it's possible to consider what you've quoted me on. This would be a very manual task, and would take some understanding of the ArcGIS REST API as well as database replication. The REST documentation can be a good starting point, but it would also be beneficial to understand some background on distributed data and sync.

The broadest description of what would need to be done is:

1. Create a replica at REST of the service you would like to duplicate in your Enterprise GDB.
2. Import the data into your Enterprise GDB.
3. Maintain the relevant replica information for future sync workflows.
4. Depending on the movement of your data, you will need to properly manage the flow of edits during sync.
   1. Is your sync one-way or bidirectional?
   2. Are records being inserted, updated, or deleted?
5. Automate the recurring sync process.

This is a very incomplete outline, as sync workflows can vary drastically from one organization to another. I would recommend looking at Distributed Collaboration before you attempt to blaze your own path. If you can't use collaboration - or a custom solution is required - then this may appeal to you.

The information is out there! It's a matter of defining what you need (data in your Enterprise GDB, etc.) and the environment/workflows that are available to do those tasks (sync, REST API, etc).

-Scott

[ViTran]

Hi Scott,

Thanks so much for your prompt reply. I did take a look at the Distributed Collaboration. It seems like a good option for us. I'll need to talk to our Database Manager about it. 

The good thing is we're considering deploying Portal for ArcGIS in our organization in a near future, so that might help to solve the problem.

Thanks again!