Email Notification if Notebook Task Fails

[Kristen]

I would like to be able to use Notebook Tasks for tasks of high importance, but I am hesitant to do so when I have to remember to go check it every once in a while to make sure the Notebook is still running properly.

When setting up a Notebook Task, there should be an option to get an automated email immediately if the task fails. This email could go to the owner of the Notebook since email is stored under the user account. It would be even better if we could choose the user or email address to send it to. Sometimes we want the whole team to know if a script fails so we send the notification to an email distribution group.

Extra points if we can customize the email to include details like the Notebook url, error details, etc.

[AntonSizo]

It would be also great to have possibility to link Webhooks to Notebook Tasks to see/react to Tasks statuses: Completed, Failed, Skipped etc.

[ChrisCoupal]

Yeah. I agree. I've had numerous occasions where my script fails because objects like feature layers that should exist, just "do not" for a time (system issue)? If this occurs 5 times, the job just stops running - no notice, no warning. Hard to have much confidence in this as a solution with it being so fragile and temperamental.

[GabrielMarcus_C]

100%  I had something fail on me for a weird reason that didn't come up in testing or any time I ran the notebook. I would like to have been notified of it via email! 

Have you seen send email with arcgis notebooks post? Others seem to get this approach to work, but for some reason I can't. 

edit: I just found this notebook from ESRI with some sample code which looks promising. 

[CandaceHankins]

I wish this was an option when setting up the task! I have a pretty lengthy script that updates data for a Dashboard and my client wants it updated two times a day. I have to manually go and check it to make sure it succeed and if it didn't, I have to manually run it. Definitely not an ideal workflow.

[RobertAnderson3]

I just discovered a task I had running failed the last two nights by complete chance, so I came looking for this solution. The above from Gabriel looks really useful but I feel like this feature should be baked in! 

Or as an alternative, ArcGIS Online has a Notifications tab (which I virtually never get anything in) maybe if a Notebook fails it should send a notification there?

[Justin_Greco]

I have a work around that has been working great for me.  I have a notebook task run every 30 minutes to check the status of the last run of all my tasks.  If that last run failed, it sends an email with the name of the task and the error message.  To prevent my tasks from entering a failed state after 5 consecutive failed runs, I am deleting that failed run after the email is sent.  

I am also working on writing the error off to hosted feature table, so I can create a dashboard of failed tasks.

I agree that it would be great to have notifications built in and integrating it with the portal web hooks is a great idea.  I also think it would be great for it to be included in ArcGIS Monitor as well.

Here is a snippet of the code I am running to check for failed task runs:

from arcgis.gis import GIS
import json
from datetime import datetime
import pytz

def getEndTime(endTime):
    utc_time = datetime.fromtimestamp(endTime / 1000)
    utc_time = utc_time.replace(tzinfo=pytz.UTC)
    eastern_time = utc_time.astimezone(pytz.timezone('America/New_York'))
    return eastern_time.strftime('%m/%d/%Y %I:%M:%S %p')
    
gis = GIS("home")
nb = gis.notebook_server[0]
##get user's tasks
task_mgr = gis.users.me.tasks
##loop through each task
for task in task_mgr.all:
    if len(task.runs) > 0:
        ##get the last run (index 0)
        lastrun = task.runs[0]
        ##check if last run failed
        if lastrun.properties.status == 'failed':
            title = task.properties.title
            title = title.replace("%20", " ")
            ##get result of last run
            if 'result' in lastrun.properties:
                result = json.loads(lastrun.properties.result)
                ##format end time to local string
                endTime = getEndTime(result['endTime'])
                if 'errors' in result:
                    error = result['errors']
                    messages = error['messages']
                    message = ''
                    ##write each line to a string with a line return
                    for m in messages:
                        message += f'{m}/n'
                    ##send the email (this is a custom function, not included)
                    sendmail(message, f'Task {title} Failed at {endTime}', 'recipent@test.com', 'sender@test.com', '')
                 ##delete the failed run
                 lastrun.delete()