Execution of a data pipeline via an API

[CarterHughes]

It would be very useful to be able to execute a data pipeline via an API. What this would enable is the ability to orchestrate them via an external system/software (or other Esri software like workflow manager), mostly namely other ETL tools that process the source data.

Our implemention utilizes Azure Synapse and there's multiple layers of data processing that occur. The most important of which arguably is serving and that's where ArcGIS Data Pipelines sits comfortably to Reverse ETL the data back into our GIS. However, it will be difficult to manage the numerous schedules for the ArcGIS Data Pipelines that are disjoint. Not to mention they will run regardless of whether prior data processing succeeded or not and that's the uncomfortable factor.

[BethanyScott]

Hey @CarterHughes ,

We have plans to support running data pipelines via the ArcGIS API for Python in a near term release. We will post here with an update when it's available.

Thank you,

Bethany

[BethanyScott]

@CarterHughesPlease see the blog linked below to learn how to run a data pipeline using ArcGIS API for Python. Please note that the offering is in "experimental" status (similar to a beta) which means it is subject to change.

https://community.esri.com/t5/data-pipelines-blog/introducing-data-pipelines-in-the-arcgis-api-for/b...

[CarterHughes]

@BethanyScott apologize for the late response! I have seen that it is available in the ArcGIS API for Python and now as an action in Power Automate, which are both great. The API must exists for both of these softwares to execute a data pipeline; however, I do not see the corresponding API endpoints documented in the ArcGIS REST APIs documentation. The idea here is more so to make the API discoverable and documented. Data engineers like myself could develop our own custom solutions with the vanilla API if so.