GeoAnalytics Data Pipeline

[MatthewMcCracken]

Hi everyone,

We’re setting up an Azure synapse pipeline where we ingest data from various ArcGIS datasets and services, convert it to Parquet files in our Data Lake where we’ll have GIS data and company data, and then create SQL views from those Parquet datasets. From there, the goal is to bring those views into our Enterprise SDE database for publishing map and feature services.

Before we go too far, I wanted to ask — is this a valid or recommended workflow? Specifically, does GeoAnalytics Engine support this type of process (ingesting GIS data, writing Parquet, and building SQL views that can integrate with ArcGIS Enterprise/SDE)?

From early research, it looks like those files can come through into our Enterprise Geodatabase, but an extra step is going to be needed for getting them in ESRI’s ST_Geometry?

Thanks for any insights or examples from folks who’ve implemented something similar!

[NoahSlocum]

Hi Matthew, here are some thoughts on this workflow:

• Ingesting GIS data - GeoAnalytics Engine supports reading from various GIS data sources, see https://developers.arcgis.com/geoanalytics/data/data-sources/ for a complete list. In this case I assume you would be reading from something like a Feature Service, or local data like a file geodatabase, both of which are supported.
• Writing Parquet - This is pretty straightforward with GeoAnalytics Engine, you could also choose to write GeoParquet. 
• Building SQL views that can integrate with ArcGIS Enterprise/SDE - You can create SQL views on parquet files stored in your Data Lake, but those SQL views can only be used within Azure Synapse, they can't be used by your Enterprise SDE database as far as I know. There are a few workflows that might be viable (in both cases, this step would be independent of writing Parquet to your Data Lake):
  • Use Geoanalytics Engine to write directly to your Enterprise SDE database using JDBC. You could then create feature and map services backed by this database.
  • Publish hosted features services directly to ArcGIS Enterprise or ArcGIS Online using GeoAnalytics Engine.

[MatthewMcCracken]

Thanks for the insight. We’re currently testing a workflow that pushes data from our Synapse SQL view into our SDE via a linked service connection. The data will remain read-only reference content, refreshed on a weekly schedule.