Data Pipelines - scheduled runs timing out

[JamesDrumm2]

In the past week a handful of our data pipelines have failed to complete and sat in a pending status for 2 hours before timing out at the 2 hour mark. We were charged 2 hours worth of credits unknowingly for each pipeline that failed to complete during this timeout issue.  Nothing was edited or changed within the pipeline and whatever caused them to timeout in the middle of the night fixed itself somehow on the next scheduled run. This leads to a few ideas for improvements:

1. The timeout limit should be much shorter than 2 hours and/or should be a flexible amount of time based on a rolling average of the previous runs.  Ex: It takes about 2-3 minuets for my pipeline to run on average for the past 3 days.  The timeout period should be set to 2 or 3x that time = 6-9 minutes or so.

2. Event logging for failed scheduled runs. Real time error logging is great, but why did my pipelines randomly get stuck in this suspended state and then seemingly randomly fix themselves?  I have no idea without a log. 

[BethanyScott]

Hi @JamesDrumm2 ,

With the latest update of ArcGIS Data Pipelines, you can now use the "Maximum run duration" parameter when creating a task. This parameter is found in the Create task modal under Advanced options and it enables you to specify how long a scheduled run should continue until it times out. Please note that the minimum duration you can specify is 15 minutes.

Thank you,

Bethany

[JamesDrumm2]

@BethanyScott 

Thank you!  We found this as soon as it was released and have implemented it on all our pipelines.  It's working exactly as we hoped it would!