Select to view content in your preferred language

Execution of a data pipeline via an API

286
2
05-29-2024 04:49 PM
Status: Open
Labels (3)
CarterHughes
New Contributor II

It would be very useful to be able to execute a data pipeline via an API. What this would enable is the ability to orchestrate them via an external system/software (or other Esri software like workflow manager), mostly namely other ETL tools that process the source data.

Our implemention utilizes Azure Synapse and there's multiple layers of data processing that occur. The most important of which arguably is serving and that's where ArcGIS Data Pipelines sits comfortably to Reverse ETL the data back into our GIS. However, it will be difficult to manage the numerous schedules for the ArcGIS Data Pipelines that are disjoint. Not to mention they will run regardless of whether prior data processing succeeded or not and that's the uncomfortable factor.

2 Comments
BethanyScott

Hey @CarterHughes ,

We have plans to support running data pipelines via the ArcGIS API for Python in a near term release. We will post here with an update when it's available.

Thank you,

Bethany

BethanyScott

@CarterHughesPlease see the blog linked below to learn how to run a data pipeline using ArcGIS API for Python. Please note that the offering is in "experimental" status (similar to a beta) which means it is subject to change.

https://community.esri.com/t5/data-pipelines-blog/introducing-data-pipelines-in-the-arcgis-api-for/b...