How to Get Email Notifications When a Feed is Down

[Jim_Wei]

Hi Velocity Team,

My team and I are trying to find the best way to receive alerts when our real-time analytics go down or when feeds are not pushing records to our RTAs. What is the best way to go about this?

I am currently trying to utilize out-of-the-box solutions rather than creating a custom Python script for this. I am guessing the Detect Gaps function and an SMTP email output is the recommended approach. Is the model I've included below something you would recommend?

Additionally, I am only interested in receiving one or two emails when my feed/RTA is down for more than x minutes/hours at a time. I don't want to receive thousands of emails showing me output records. Any advice on how to achieve this would be greatly appreciated. Thank you.

 @Yini_Song @PeterNasuti 

Jim

[PeterNasuti · Accepted solution]

@Jim_Wei If you ever witness a feed or analytic go down and stop functioning as expected, please make sure to log an Esri Support Services ticket so the root cause can be addressed by the product team.

A combination of Map Fields to designate the relevant Track ID field with Detect Gaps > Control Event Volume can be used to limit the number of emails sent in the event of features coming through that pipeline.

The ArcGIS Velocity product team is committed to continuing to enhance product level notifications and alerts and we will take this request under consideration. In the meantime, a feed configured on your output feature layer with Detect Gaps > Control Event Volume in a separate real-time analytic would achieve the core goal of automatically notifying if features are not flowing as desired and be contained in separate elements from your processing analytic.

[JeffSilberberg]

@Jim_Wei

    I do this from the source of the data so I know if Velocity is properly ingesting data.  If the Queue of transactions targeted to Velocity backs up then we trigger an alarm using Nagios and retrigger if the condition does not clear every n minutes, in some cases with an escalation to more / different contacts.  

   Another approach would be to push a keep alive transaction every n minutes into the feed and check that it comes back to you. 

   ---------------

[Teresa_Blader]

How did this turn out? I am currently trying to do this with Air Quality sensors. I'd like to push an email when a sensor hasn't had detection within an 1 hour indicating the sensor disconnected from Wifi or something. Detect Gaps seems to be the logical step, but sounds like it would send an email each time is tries which wouldn't help.

Additionally we use M365 which no longer supports SMTP in our organization, so now I don't know how to do that either. There does not appear to be an oAuth option.

Maybe creating a second analytic could take the Detect Gaps layer and pass IsGap = true and GapDuration to the email. 

[JeffSilberberg]

@Teresa_Blader 

 As Peter suggested above a combination of Detect Gaps and Control Event Volume Nodes will probably get you to the throttle point you want, although I am struggling a little to get the Analytic to behave differently during different hours of the day.   

As far as SMTP out goes, the new Amazon outputs supported in 4.2 may help a lot with working around that, or maybe find an alternate E-Mail forwarder such as -  https://improvmx.com/  

Note: Our pre-processor which also does a lot of refactoring of the JSON for the Feed, has a number of pre-Velocity alarm points built in as well. 

[Jim_Wei]

@JeffSilberberg Thanks for your response. I think my team would've considered polling feature services and RTAs using Python scripts to check status of the feed and analytics. We were trying to do everything using only ArcGIS Velocity and OOTB. 

@Teresa_Blader We did end up using the detect gap tool and SMTP email output. Our notification mode for detect gaps was set to "on change". This means we would only get notifications when it first detects a gap, and wouldn't send us tons of emails. Using that and 'Filter by Expression' custom logic, you can probably build something close to what you are looking for.