Improve Scheduling of Tasks

[KevinWyckoff1]

It is great that scheduling is available for tasks. However, the available options are very limiting.

I have a workflow setup to update a feature service from our work management system by calling a GeoJSON URL, and using the resulting data to overwrite the feature service. It needs to run often during business hours, roughly 8 am to 4 pm Monday through Thursday. I initially set it up to run every 15 minutes, but could not make it self pause at night. So I let it go, and I ate through a ton of credits unnecessarily. So I am now manually enabling the task at the start of the work day and pausing it at the end.

There needs to be a better way to schedule runs with more granularity. I would love if Data Pipelines could just copy the options available in Windows Task Scheduler.

[MichaelVolz]

Can you not just use Windows Task Scheduler instead since it has better options 

[KevinWyckoff1]

@MichaelVolz I could use Windows Task Scheduler if Data Pipelines was an application installed on my server. Data Pipelines is not available as part of ArcGIS Enterprise, and I can't access ESRI's servers so no way to do that. Although it would be very cool to include Data Pipelines in future Enterprise releases.

Alternately if I could create a workflow in Data Pipelines and export it as a Python Script, I absolutely could pull that into my Enterprise environment and run on my server using Windows Task Scheduler.

Your comment points to the double edged sword of using the ArcGIS Online Platform as a Service, you lose some really basic functionality found on on-premise applications.

[JakeSkinner]

Hi @KevinWyckoff1,

To build off of @MichaelVolz's suggestion, you can easily accomplish this with Windows Task Scheduler and ArcGIS Pro.  Create a python script that performs the following operations using arcpy:

• convert the GeoJSON to a feature class using the JSON to Features tool
• truncate the feature service
• append the feature class to the service

[MaxPayson]

Hi @KevinWyckoff1,

Thank you for the idea and taking the time to submit it.

Another approach for now is to use the Python API to create the task for the data pipeline. This provides more granularity by supporting cron expressions. Once created, you will be able to use the task like other tasks in the data pipelines web app - you can view it, stop it and start it, review the results, etc. The interval properties just may not populate correctly if you try to edit them.

Here's a blog post with more information. Also, here's an example Python script. Note that the expression is in Coordinated Universal Time (UTC), so you may need to adjust the days and times for your time zone. For example, the East Coast is currently 4 hours behind UTC, so the hours would be 12-20.

from arcgis.gis import GIS
gis = GIS("home")  # ArcGIS Online Notebook initialization
pipeline = gis.content.get("{data pipeline item ID}")
gis.users.me.tasks.create(pipeline, "*/15 8-16 ? * 1-4", "RunDataPipeline", title="Daily workday updates")

# CRON explanation
# */15 = every 15 minutes
# 8-16 = 8am to 4pm UTC
# ? = every day of the month
# * = every month
# 1-4 = Monday (1) to Thursday (4)

[JonathanMcD]

The solution would be simple, all depends on how open ESRI are to implementing it. There is the Week option which allows you to pick the days, there's a time selection, if only the start time also allowed the runs every "15 mins" picker. Then either a setting for number of times or until end time.