The ArcGIS Data Pipelines Connector in Power Automate is great! I am certainly going to implement it. However, it is a band-aid solution for users like myself who execute data engineering at the enterprise scale and use Azure Data Factory / Azure Synapse / Microsoft Fabric for orchestration (and more). Power Automate was a great first step but I cannot version control it nor have it as robustly tested nor publish changes across environments like I can for my Azure Synapse environment for example.
I believe this is also a great idea for implementation because it would help delegate some of the more complex aspects of data engineering like orchestration and even logging to other third parties like Microsoft. There are other open ideas requesting some of this functionality already like logging. Customers who need that can use an orchestration tool from another third party and allow Data Pipelines focus solely on geospatial data engineering within the ArcGIS ecosystem.
Well said — I agree with this completely. The Power Automate connector is a useful starting point, but for enterprise-scale data engineering it doesn’t offer the same robustness you get with orchestration platforms like Azure Data Factory, Synapse, or Microsoft Fabric.
Being able to integrate Data Pipelines with those tools would bring huge benefits: proper version control, promotion across environments, robust testing, and enterprise-grade logging/orchestration. That would let Esri focus on the geospatial data engineering strengths, while organizations leverage existing enterprise orchestration platforms for scheduling, monitoring, and governance.
+1 to this idea — tighter integration with enterprise orchestration tools would make Data Pipelines much more powerful in large-scale environments.
Regards,
Venkat
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.