Email warnings when Velocity feed fails

DrewDowling

Is there any way to configure email warnings when a Velocity feed fails? We have a HTTP poller feed and would like email warnings if the source becomes unavailable. Is this something we can configure?

The email warnings we get when a feed exceeds it 25KB limit is helpful. We would ideally like something like this if the HTTP poller get request fails.

MichaelVolz

Is this link helpful?

https://community.esri.com/t5/arcgis-velocity-questions/how-to-get-email-notifications-when-a-feed-i...

DrewDowling

Thank you Michael. The link is very informative but unfortunately unhelpful in our case.

The detect gaps tool doesn't work on polygon data
99% of the time there is no data to pull, the source has no current active polygons.

One user suggested using scripts to test the source. I will look into using a notebook script to test the response of the feed source, but then the question becomes: how do I get a warning if the notebook script fails to run?

So the ideal solution would be a warning from the feed itself.

JeffSilberberg

The answer in today's environment is No, BUT.

I have a Feed to which I inject a WatchDog transaction once every hour, and check that the Watchdog comes back to me from the Real Time Analytic that processes the Feed via an HTML POST action. We check the round trip and trigger SMS/E-mail alerts if we do not get a full circle in n minutes.

Since you are polling, not posting, you would need to be able to inject the Watchdog transaction into the target of the Polling transaction.

DrewDowling

The feed we are pulling from is a 3rd party feed we have no control over, does this mean the approach won't work?

I'm unfamiliar with WatchDog transactions but it's an approach I'd like to know more about if you think it will help in my situation.

JeffSilberberg

Well,

Again, it depends. If your 3rd party feed has a transaction that is consistent and processes every n minutes, you could use that as your Watchdog; if not, then this approach will not work for you.

Sorry.

PeterNasuti

Hi @DrewDowling This is in the longer term product plan to offer "rate drop or failure" notifications as part of Velocity feeds and analytics to notify users if their upstream systems fail to send data or if there is any run status change (like a feed or analytic in a failure state).

DrewDowling

Thanks you for the update @PeterNasuti

DrewDowling

In the end we created a notebook script that runs as a scheduled task. The script uses the AGOL rest api to check velocity status and errors. It then emails group members if it finds an issue. Setting it to run at 15 minute intervals consumes approx 2.5 credits a day.

Here's a copy of the script in case someone finds it useful.

from arcgis.gis import GIS
gis = GIS("home")

import requests
import json


def emailWarning(errormessage):
    emengencyMapGroupID = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'  # Replace with your actual group ID
    # Get the group object using the group ID
    emengencyMapGroup = gis.groups.get(emengencyMapGroupID)
    emailSubject = 'Test Errors recorded in the Genasys Velocity Feeds Test'
    # Users have to be in the group to receive the email
    emengencyMapGroup.notify(users=['OES_Dev'], subject=emailSubject, message=errormessage)


# Check that the source data wfs endpoint is up and running
url='https://source.feed.com/geoserver/z/wfs'
payload={'authkey':'mykey',
'SERVICE':'WFS',
'REQUEST':'GetFeature',
'VERSION':'2.0.0',
'TYPENAMES':'z:evacuation_zone',
'srsName':'EPSG:4326',
'outputFormat':'application/json',
'CQL_FILTER': 'zone_status<>\'Normal\''}
r = requests.get(url, params=payload)

print(r.status_code)
# Check if the request was successful
if (r.status_code != 200): 
    emailWarning('Failed to connect to the source WFS endpoint. Confirm source is up and running')


# Check the status of the Velocity feeds
# Get the Velocity feeds
velocity = gis.velocity
# velocity
feeds = gis.velocity.feeds
# feeds
Sample_feed = feeds.get("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
status = Sample_feed.status
# print(status)
print(status['status'])
# Check if the feed is started
if(status['status'] != 'started'):
     emailWarning('MY poller ingest feed is not started. Please log in to Velocity and restart it.')


sample_metric = Sample_feed.metrics
# print(sample_metric)
print(sample_metric['numErrors'])
# Check if there are any errors in the feed metrics
if sample_metric['numErrors'] > 0:
    emailWarning('MY poller ingest feed is recording errors. Please log in and investigate.')


# Check the status of the real-time analytics
# Get the real-time analytics
real_time_analyitics= gis.velocity.realtime_analytics
real_time_analyitics
event_inget_analytic =real_time_analyitics.get('XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')
event_inget_analytic.status
print( event_inget_analytic.status['status'])
# Check if the real-time analytics process is started
if event_inget_analytic.status['status'] != 'started':
    emailWarning('My Real time Analytic processes is not started. Please log in to Velocity and check the status. ')

# Check if there are any errors in the real-time analytics metrics
for value in event_inget_analytic.metrics.items():
    # metrics are in a list of dictionaries
    if( type(value[1])==list):
        if(len(value[1])> 0):
            for dic in value[1]:
                # Check if the dictionary has an 'errorCount' key
                if 'errorCount' in dic:
                    if dic['errorCount'] > 0:
                        emailWarning('One of the My Real time Analytic processes had an error. Please log in to Velocity and check the logs. ')
                        print('we have a problem')