idea Run a data pipeline on a schedule in Data Pipelines Ideas

Run a data pipeline on a schedule

sjones_esriau — Thu, 15 Jun 2023 09:24:05 GMT

For similar reasons that we can run Notebooks on a schedule, it would be advantageous to run Data Pipeline's on schedules too.

Or alternatively run a pipeline from a scheduled notebook or even export a pipeline to python for use in a notebook.

Re: Run a data pipeline on a schedule

BruceHarold — Thu, 15 Jun 2023 14:04:00 GMT

Hi Sean, thanks for the suggestion, scheduling pipelines is planned and targeted for the next release.

If you have more suggestions please add them here too.

Re: Run a data pipeline on a schedule - Status changed to: In Product Plan

BruceHarold — Thu, 15 Jun 2023 13:56:13 GMT

Re: Run a data pipeline on a schedule

WilliamMayr — Fri, 16 Jun 2023 14:52:43 GMT

This was my first thought as well. Glad to hear scheduling is coming soon! This (data pipelines) is awesome functionality and will change how our org works with ArcGIS.

Re: Run a data pipeline on a schedule

WilliamMayr — Fri, 23 Jun 2023 14:29:55 GMT

@BruceHarold After talking to a couple of our analysts, it was asked if it would be possible to trigger the run of one of these pipelines with a REST web service call. The use case they had in mind is an external application prompting end-users for information, running a bunch of data science code based on these inputs, spitting the output of models out to Azure, and then using this data pipeline to read it from ADLS2 and refresh the feature layer on-demand.

We would have use cases for both scheduled and ad-hoc scenarios.

Re: Run a data pipeline on a schedule

BruceHarold — Fri, 23 Jun 2023 14:47:39 GMT

Thanks William, I can advise a REST call to run a pipeline is under consideration.

Re: Run a data pipeline on a schedule

RobertAkroyd1 — Wed, 13 Sep 2023 07:43:56 GMT

Hi @BruceHarold Data Pipelines will be a great feature once fully released and more capable. Having the ability to run a pipeline via a REST API call would be great as would open up a load of integration options, including running a pipeline from the new ArcGIS Online Workflow Manager.

Re: Run a data pipeline on a schedule

BruceHarold — Wed, 13 Sep 2023 13:34:58 GMT

@RobertAkroyd @RobB It's coming! Thanks for the feedback.

Re: Run a data pipeline on a schedule

BethanyScott — Thu, 26 Oct 2023 16:14:48 GMT

Hi@sjones_esriau @WilliamMayr @RobertAkroyd1,

With the October 2023 update of ArcGIS Online, the ability to schedule data pipelines is now available.

To learn more, check out the What's New in Data Pipelines (October 2023) blog.

Try it out and let us know how it goes, or if you have any questions leave a comment here and the Data Pipelines team will help you out 🙂

Thank you,

Bethany

Re: Run a data pipeline on a schedule - Status changed to: Implemented

Sarah_Hanson — Wed, 08 Nov 2023 20:17:37 GMT