Execution of a data pipeline via an API

CarterHughes

It would be very useful to be able to execute a data pipeline via an API. What this would enable is the ability to orchestrate them via an external system/software (or other Esri software like workflow manager), mostly namely other ETL tools that process the source data.

Our implemention utilizes Azure Synapse and there's multiple layers of data processing that occur. The most important of which arguably is serving and that's where ArcGIS Data Pipelines sits comfortably to Reverse ETL the data back into our GIS. However, it will be difficult to manage the numerous schedules for the ArcGIS Data Pipelines that are disjoint. Not to mention they will run regardless of whether prior data processing succeeded or not and that's the uncomfortable factor.

CarterHughes

@BethanyScott apologize for the late response! I have seen that it is available in the ArcGIS API for Python and now as an action in Power Automate, which are both great. The API must exists for both of these softwares to execute a data pipeline; however, I do not see the corresponding API endpoints documented in the ArcGIS REST APIs documentation. The idea here is more so to make the API discoverable and documented. Data engineers like myself could develop our own custom solutions with the vanilla API if so.