Using Data Interoperability to sync data to ArcGIS Online
What do you do when you've found something so beautiful that you have to have a copy of it? Turn to ArcGIS Pro and the Data Interoperability extension, of course! The Data Interoperability extension (from here on, shortened to Data Interop) is a very powerful tool for ArcGIS Pro and can be used to simplify your data ETL (Extract, Transform, Load) processes across the ArcGIS platform. The true power of Data Interop is the ability to work within a ModelBuilder like environment, meaning that you do not need to write any code to work with over 400 different data formats. Chances are that if you are working with an obscure datatype, it is supported – see the 400 + supported formats here.
I’m not going to go into detail with obscure data formats, rather, I'm going to highlight a very common workflow that many of the organizations I work with can take advantage of – syncing data from an Enterprise geodatabase (GDB, aka ArcSDE) to hosted feature layers (ArcGIS Online or ArcGIS Enterprise). This is very advantageous in working with Survey123, because Survey123 does not support writing to traditional versioned datasets, and generally works best with hosted feature layers.
WHY DATA INTEROPERABILITY
A common application where we might deploy this type of workflow would be for hydrant inspections. Many organizations use Survey123 with their hydrant inspections because of the simplicity of this solution, and it supports adding related records for keeping an archive of inspection data. If your hydrants are in a versioned feature class, then Survey123 cannot work with it. We can manually export the hydrants from our GDB, but of course they will become outdated over time. This workflow will keep your versioned (or unversioned) GDB data in sync with your hosted feature layer(s) in ArcGIS Online. Note that this workflow works just as well with a file GDB, although let’s have a more serious discussion if you are using personal GDBs or shapefiles as a source for your authoritative data.
Another common application for using Data Interoperability is keeping ArcGIS Online Open Data in sync with your authoritative data. If your Open Data originates from non-GIS data sets, this is more of a no-brainer, and to make things easier to share with the public, ArcGIS Online supports standalone tables without any geography.
ALTERNATIVE TO DATA INTEROPERABILTY
I recommend that your first option to sync data to ArcGIS Online is to use Distributed Collaboration, which is supported at ArcGIS Enterprise 10.5.1 and later. Benefits are that the scheduling is automatically setup on your production Enterprise environment, and you can sync on demand. However, there are some requirements which may not be an option for some organizations. First, you will need to have access to an ArcGIS Enterprise Base Deployment (10.5.1 or later). You will also need to enable the sync capability on your Enterprise feature service. If this workflow works for you and your data, then stop reading and finally start that 3D building model of your city that you always dreamed of.
If this workflow does not work for you, then I suggest firing up Data Interop with ArcGIS Pro. I have attached a sample workspace (.fmw file) that is embedded within an ArcGIS Pro Package to this post and a couple of introductory videos explaining the process. Data Interop workspaces can be imported into a Toolbox as a Spatial ETL tool, and new at ArcGIS Pro 2.5, scheduled to run on the frequency and recurrence of your choice (hourly, daily, weekly, every other week, etc…).
HOW
The Workspace is quite simple, and I will discuss the important aspects. There are two readers, three transformers, and one writer. The first reader connects to ArcGIS Online to read the hosted feature layer that I originally published from the file GDB, and the second reader connects to the file GDB (our authoritative dataset, and which will more commonly be residing in an Enterprise GDB or ArcSDE). The transformers are an AttributeManager, a DateTimeConverter, and a ChangeDetector. The ChangeDetector is straightforward, we instruct it to detect changes among attributes, geometry, or both.
Data Interoperability Sync Workspace (2 Readers, 3 Transformers, 1 Writer):
The DateTimeConverter formats the file GDB dates to be a compatible format because the original dates in the file GDB are being read without timestamps (“20080903” aka September 3, 2008), and the ArcGIS Online reader is formatting the data with timestamps (“20080903000000” aka September 3, 2008 @ 12 AM). The ChangeDetector treats these dates as different, so we use the DateTimeConverter to get the format correct (%Y%m%d%H%M%S shown as: 20080805000000 vs 20080805). Your dates and times may not need formatting, but if so, the DateTimeConverter is very useful – you can also add or subtract hours with the DateTimeCalculator if your times are in different zones (ex: UTC vs Central Time Zone).
The AttributeManager transformer is being used to change the ArcGIS Online GlobalID to UPPER case as it is being read in as lower case (“{@UpperCase(@Value(GlobalID))}”). Finally, we have an ArcGIS Online Writer which takes the output from the Updated, Inserted, and Deleted ports and uses this information to Add, Delete, or Update the corresponding features in ArcGIS Online. This Writer needs the GlobalID from the file GDB in order to process Updates and Deletes. There is a flag called "fme_db_operation", which the ChangeDetector sets to instruct the Online writer which operation to perform.
This workflow becomes more beneficial as we increase the size of our dataset. However, if you are writing a smaller dataset and don’t want to spend the processing power determining which features have been updated, inserted, or deleted, there is a much simpler option which is to overwrite all features. This depends on the size and/or number of features in your dataset, and the time it takes to re-write these features to ArcGIS Online. If you have a large dataset containing many thousands of features, or polylines and polygons which contain many vertices, then you might not want to spend the time it takes to delete and recreate all features. By only writing changes, the process will be much quicker and lighter on Internet bandwidth. The less data we are sending over the Internet, the more reliable our ETL process will ultimately be. If overwriting all features works for your workflow, you can get rid of the ArcGIS Online Reader (no need to compare as we will be overwriting all data), the three transformers, and change the settings on the writer to “INSERT” and “Truncate First: Yes”. These settings will delete all features in the hosted feature layer and perform an append from all features in your source dataset, effectively recreating the dataset. This is considered a heavy online process in that it copies all data, not just the changed data.
For those of you who remember any detail from the 2nd paragraph may recall that I discussed syncing data from an Enterprise GDB, not a file GDB. What gives? Well, there is not much of a hurdle changing the input from a file GDB to an Enterprise GDB and the underlying process is the same. The reason I chose the file GDB is to share the data with you. To convert the Data Interoperability workspace to an Enterprise GDB, all you have to do is add the “Esri Geodatabase (ArcSDE Geodb)” reader and configure it with your database connection file (the .sde file you use in ArcMap or Pro to connect to and edit your data).
Below you will find an ArcGIS Pro Project Package (.ppkx) that has been created for ArcGIS Pro 2.7. When you open the Data Interop Workspace (GDBToArcGISOnline), the file GDB location will still be accurate because it is using the relative path from a User Parameter, however, you will have to replace my ArcGIS Online connections (both the Reader and Writer) with ones you have access to publish to. You will also have to publish the Hydrants feature layer that shipped with the Pro Package and loads with the map. Keep the schema the same and add a new ArcGIS Online reader to point to the new Hydrants dataset you just published.
That’s it, happy ETL'ing! Other basics if you have not used the Data Interoperability Extension before:
- Data Interoperability licenses do not come with ArcGIS Pro and need to be purchased separately. Your organization may or may not already be licensed for this extension.
- Data Interoperability is a separate install, so it needs to be installed on top of ArcGIS Pro. You can assign the licenses from ArcGIS Online or ArcGIS Enterprise. Download the extension installation package from My Esri.
- You can incorporate a Spatial ETL tool with ModelBuilder. You can have geoprocessing tools pass data over to a Data Interoperability workspace, and vice versa in a ModelBuilder model.
- There is a free Training Seminar located here, which will highlight the many different ways you can use this extension.
Data Interoperability Licensing and ArcGIS Pro Location Basics:
Diving into the Workspace settings and automating the tool:
## Updated the ArcGIS Pro Project Package to Pro 2.7 and changed the Spatial ETL tool to relative path so you do not have to re-point the path.