Run a data pipeline on a schedule

[sjones_esriau]

For similar reasons that we can run Notebooks on a schedule, it would be advantageous to run Data Pipeline's on schedules too.

Or alternatively run a pipeline from a scheduled notebook or even export a pipeline to python for use in a notebook.

[BruceHarold]

Hi Sean, thanks for the suggestion, scheduling pipelines is planned and targeted for the next release.

If you have more suggestions please add them here too.

[WilliamMayr]

This was my first thought as well. Glad to hear scheduling is coming soon! This (data pipelines) is awesome functionality and will change how our org works with ArcGIS.

[WilliamMayr]

@BruceHarold After talking to a couple of our analysts, it was asked if it would be possible to trigger the run of one of these pipelines with a REST web service call. The use case they had in mind is an external application prompting end-users for information, running a bunch of data science code based on these inputs, spitting the output of models out to Azure, and then using this data pipeline to read it from ADLS2 and refresh the feature layer on-demand.

We would have use cases for both scheduled and ad-hoc scenarios.

[BruceHarold]

Thanks William, I can advise a REST call to run a pipeline is under consideration.

[RobertAkroyd1]

Hi @BruceHarold Data Pipelines will be a great feature once fully released and more capable.  Having the ability to run a pipeline via a REST API call would be great as would open up a load of integration options, including running a pipeline from the new ArcGIS Online Workflow Manager.

[BruceHarold]

@RobertAkroyd @RobB  It's coming!  Thanks for the feedback.

[BethanyScott]

Hi@sjones_esriau @WilliamMayr @RobertAkroyd1, 

With the October 2023 update of ArcGIS Online, the ability to schedule data pipelines is now available. 

To learn more, check out the What's New in Data Pipelines (October 2023) blog.

Try it out and let us know how it goes, or if you have any questions leave a comment here and the Data Pipelines team will help you out 🙂 

Thank you,

Bethany