Creating a Sync of Enterprise Database on ArcGIS Datastore

[DarrylKlassen1]

We currently have an Enterprise SDE database (Oracle) that houses our GIS data.  In the past we have simply created map/feature services off of this and consumed them in the AGOL world.  Now we are implementing ArcGIS Enterprise Portal and have an ArcGIS Datastore Setup.  We were wondering if it is possible to set something up to pull data out of our Oracle database and put it into the ArcGIS Datastore for consumption, so when these services are being consumed, it is not putting stress on our Enterprise database?  These serveices can be Read-only, so we can update them from our Enterprise database periodically.  So baiscally:

1. How can we make a copy of our Enterprise database data on ArcGIS Datastore

2. How can we update this data periodically to have a refresh?

3. Is this even a good practice/idea?  

I have been attempting to export our Enterprise database data in smaller chucks out to File GDBs and then uploading them to the ArcGIS enterprise portal and publishing them, but when I attempt to script the overwrite, i get this error:

Perhaps there is a better way of doing this.  Thanks.

[MarceloMarques]

ArcGIS Datastore does not support all the functionality that an Oracle Enterprise Geodatabase supports.

What is ArcGIS Data Store?—ArcGIS Enterprise | Documentation for ArcGIS Enterprise

If using only simple featureclasses without any advanced geodatabase functionality then you can move the data to the Datastore when you publish the service with the option to copy the data to server, read more in the documentation. If you publish as a feature service, then you can edit the data using the service.

Data and publishing in ArcGIS Enterprise—Portal for ArcGIS | Documentation for ArcGIS Enterprise

[DarrylKlassen1]

Thanks for the info Marcelo.  I am able to publish simple feature classes into my datastore without issue, and this fine that it doesn't have the same advanced geodatabase functionality.  Is there a way to automatically (script??) update this data on a regular basis though?  So when my data changes in my Oracle Database, I can update it in my data store? 

[MarceloMarques]

You will need to write a custom arcpy script or use ArcGIS Model Builder or ArcGIS Data Interoperability Extension or Safe Software FME Workbench or FME Sever.

[DarrylKlassen1]

Does ESRI have any examples on how to do this?  Example of how to set up this workflow? Txs

[MarceloMarques]

https://www.google.com/search?q=arcpy+update+hosted+feature+layer

https://developers.arcgis.com/python/samples/updating-features-in-a-feature-layer/

https://community.esri.com/t5/arcgis-api-for-python-questions/updating-feature-layer-using-arcgis-ap...

[ArchitSrivastava]

Hello @DarrylKlassen1,

As mentioned by @MarceloMarques, I also agree that there is no direct way to do this except a script or an FME Desktop/Server workbench. However, I do have some suggestions to add to this which are as follows including some considerations:

• The access point for ArcGIS DataStore would always be a Feature service in your scenario. I would recommend refraining from directly connection to Relational Datastore of ArcGIS DataStore.
• The output service is read-only
• Is this even a good practice/idea? : Not sure, but this is an interesting approach, made me think of some new components to work this out with script.

A possible suggested workflow: Use Python and Windows Task Scheduler  

Requirements :

• Python/ArcGIS API for Python (arcpy module)
• cx_Oracle (for connecting to Oracle Database)
• Any other modules depending on approach to detect changes

Steps

1. Database Connection
   1. Connect to your Oracle Database using cx_Oracle. You will need to provide the database connection details and credentials.
2. Detect Changes
   1. You will need to implement logic to detect changes in your Oracle Database. This can be done by comparing the current state of the data with a previous state. You can use timestamps, unique identifiers, or other criteria to identify changes.
3. If Changes Detected
   1. If changes are detected, proceed with the following steps:

      1. Update Feature Service

      2. Use the ArcGIS API for Python (arcpy) to connect to your ArcGIS Server and feature service. You can use the arcpy.mapping.UpdateLayer method to update the feature service.

      3. Overwrite Feature Service

      4. If you are using ArcGIS Online or Portal for ArcGIS, you can use the arcpy.mp.UpdateWebLayerSharing method to overwrite the existing feature service with the updated data.

4. Scheduling and Automation

   1. You mentioned that you want this script to run at scheduled intervals. You can use scheduling tools such as  Task Scheduler (on Windows) to run your Python script at specific times or intervals.

5. Logging and Error Handling

   1. Implement proper logging and error handling in your script to track its execution and handle any issues that may arise.

Please note that the implementation details will depend on the specifics of your Oracle Database schema, how you want to detect changes, and how you have published your feature service. You may need to adapt the script to your organization's needs.

Additionally, you would also need to take following under consideration

• How often the data is being updated based on this the frequency of the script in Task Scheduler
• If at any point of time, the feature service becomes editable. This would bring more complexity to the workflow as we would have to detect changes both ways and that would be a tricky situation
• Make sure the name of the datasets or the attributes within the dataset does not changes of if it does the script needs to be adjusted accordingly.

Let me know if this clarifies things.

Hope if helps!

-Archit