Send Email When a Feature is Added to an ArcGIS Online Hosted Feature Service

49064
176
06-02-2017 05:46 AM

Send Email When a Feature is Added to an ArcGIS Online Hosted Feature Service

With ArcGIS GeoEvent it is very easy to setup a notification when a new feature is added to an ArcGIS Online Hosted Feature Service.  Instructions can be found here.  If you do not have GeoEvent you can still accomplish this using Python.  Below is an example on how to do this. 

When editor tracking is enabled, new features are recorded in a field called 'CreationDate'.  The script queries this field within the service and compares the time to the current time (minus one hour).  If the time is greater than the current time, it will send an e-mail with a specified field's values for the new features.

Windows Task Scheduler can be setup to execute this script at a given interval (i.e. every hour).  This will report any new features within that last hour.

The below example is written for the Citizen Problem Report application.  It utilizes 2 fields (probtype & status) when querying the service.  These variables can be commented out, or updated to reference different fields, if working with a different service:

JakeSkinner_4-1627925313899.png

 

The SUBJECT and TEXT variables in the sendEmail function can be updated to whatever you would like to include in the e-mail:

JakeSkinner_5-1627925339918.png

 

import requests, json, datetime, time, smtplib
from datetime import timedelta
from email.mime.text import MIMEText

# Disable warnings
requests.packages.urllib3.disable_warnings()

# Variables
username = 'jskinner_CountySandbox'                                                                                   # AGOL Username
password = '*******'                                                                                                  # AGOL Password

URL = 'https://services.arcgis.com/dlFJXQQtlWFB4qUk/arcgis/rest/services/CitizenProblems/FeatureServer/0'             # Feature Service URL
dateField = 'CreationDate'                                                                                            # Date field to query
hoursValue = 1                                                                                                        # Number of hours to check when a feature was added

fromEmail = 'no-reply@esri.com'                                                                                       # Email sender
toEmail = ['user1@esri.com', 'user2@esri.com']                                                                        # Email receiver(s)
smtpServer = 'smtp.esri.com'                                                                                          # SMTP Server Name
portNumber = 25                                                                                                       # SMTP Server port


# Function to send email
def sendEmail():
    SUBJECT = 'Problem Reported'
    TEXT = "{0} was created with status {1}".format(typeProb, status)
    smtpObj = smtplib.SMTP(host=smtpServer, port=portNumber)

    msg = MIMEText(TEXT)
    msg['Subject'] = SUBJECT
    msg['From'] = fromEmail
    msg['To'] = ", ".join(toEmail)

    smtpObj.sendmail(fromEmail, toEmail, msg.as_string())
    print("Successfully sent email")
    smtpObj.quit()

# Generate AGOL token
try:
        print('Generating Token')
        tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
        params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'}
        r = requests.post(tokenURL, data=params, verify=False)
        response = json.loads(r.content)
        token = response['token']
except:
        token = ''

# Return largest ObjectID
whereClause = '1=1'
params = {'where': whereClause, 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
r = requests.post(URL + '/query', data = params, verify = False)
response = json.loads(r.content)
try:
    response['objectIds'].sort()
except Exception as e:
    print("Error: {0}".format(e))

count = len(response['objectIds'])


# Query service and check if CreationDate time is within the last hour
if count < 1000:
    params = {'f': 'pjson', 'where': "1=1", 'outFields' : '*', 'returnGeometry' : 'false', 'token' : token}
    r = requests.post(URL + '/query', data=params, verify=False)
    response = json.loads(r.content)
    for feat in response['features']:
        typeProb = feat['attributes']['probtype']
        status = feat['attributes']['status']
        createDate = feat['attributes'][dateField]
        createDate = int(str(createDate)[0:-3])
        t = datetime.datetime.now() - timedelta(hours=hoursValue)
        t = time.mktime(t.timetuple())
        if createDate > t:
           sendEmail()
else:
    y = minOID
    x = minOID + 1000

    ids = response['objectIds']
    newIteration = (math.ceil(iteration/1000.0) * 1000)
    while y < newIteration:
        if x > int(newIteration):
            x = newIteration
        where = OID + '>' + str(y) + ' AND ' + OID + '<=' + str(x)
        print('Querying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
        params = {'f': 'pjson', 'where': where, 'outFields' : '*', 'returnGeometry' : 'false', 'token' : token}
        r = requests.post(URL + '/query', data=params, verify=False)
        response = json.loads(r.content)
        for feat in response['features']:
            typeProb = feat['attributes']['probtype']
            status = feat['attributes']['status']
            createDate = feat['attributes'][dateField]
            createDate = int(str(createDate)[0:-3])
            t = datetime.datetime.now() - timedelta(hours=hoursValue)
            t = time.mktime(t.timetuple())
            if createDate > t:
                sendEmail()
        x += 1000
        y += 1000

 

Comments

How should you adjust the log-in for a Portal log-in? 

Williams, CarolAnn‌ you would have to adjust how the token is generated:

tokenURL = 'https://portal.esri.com:7443/arcgis/sharing/rest/generateToken/'
params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'https://portal.esri.com'}
req = urllib2.Request(tokenURL, urllib.urlencode(params))
try:
    response = urllib2.urlopen(req)
except:
    gcontext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
    response = urllib2.urlopen(req, context=gcontext)
data = json.load(response)
token = data['token']

Jake Skinner

I'm receiving an error 

Traceback (most recent call last):
File "C:\Python27\ArcGIS10.7\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 326, in RunScript
exec codeObject in __main__.__dict__
File "K:\Scripts_Python\SanitaryFeatureUpdatedEmail.py", line 84, in <module>
createDate = feat['attributes'][dateField]
KeyError: 'created_date'

My service is hosted (on my portal) and has a creation date field called "created_date" with alias "CreationDate", but it is not being pulled in the url response. 

I printed data (json.load(response)) and got a list of fields and attributes that are associated with the layer, but nothing related to date or creator. 

{u'hasM': False, u'features': [{u'attributes': {u'issue': u'Abandon'}}], u'fields': [{u'alias': u'Issue', u'length': 50, u'type': u'esriFieldTypeString', u'name': u'issue'}], u'globalIdFieldName': u'globalid', u'hasZ': False, u'exceededTransferLimit': False, u'objectIdFieldName': u'objectid'}

Has anyone else run into this issue? 

I tried 'outfields' : '*', to see if I could pull more fields, but it didn't make a difference in what was printed for data. 

Thank you for any advice/ troubleshooting ideas. 

Just FYI:

changing the 'outfields' to 'outFields' worked. I was just browsing this: Query (Feature Service/Layer)—ArcGIS REST API: Services Directory | ArcGIS for Developers when I thought I'd try it. outfields returned a limited set of the fields, whereas outFields returned them all. 

Need help with following error (using Python 2.7.8):

Generating Token

Traceback (most recent call last):
  File "C:\Users\jlydon\Documents\complaint_notification.py", line 36, in <module>
    response = urllib2.urlopen(req)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 404, in open
    response = self._open(req, data)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 422, in _open
    '_open', req)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>
>>>

Immediate research suggested installing OpenSSL??

Jake Skinner

I am getting the same error as Mary.

I am trying to send an email via outlook 

Traceback (most recent call last):
File "C:\______\sendEmail_py", line 44, in <module>
for feat in data['features']:
KeyError: 'features'

I am sure my name is spelled correctly and I did print my token and I got a long string.

Rickey Fite‌ the data object appears to not be returning anything.  You can test this by adding a print statement before line 44.  Ex:

data = json.load(response)
print(data)
for feat in data['features']:

Anthony Whitley‌ can you access the following URL in a web browser on the server you're running the script:

https://www.arcgis.com/sharing/rest/generateToken

This is what printed 

{u'error': {u'message': u'Invalid Token', u'code': 498, u'details': []}}

Rickey Fite‌ can you share the service to an ArcGIS Online Group and invite my user account (jskinner_CountySandbox)?

Jake Skinner

Done,

The layer is just a test and it is called Cougar_sightings. I am will be using a different layer when I get it working. 

Ultimately I would like to have the script work with portal.

Rickey Fite‌ I forgot to update the code above.  Plainfield GIS‌ found the issue to be the outfields parameter.  This should be outFields. I went ahead and updated the code above.  This will return the data successfully with the query.

Jake Skinner

I now do not have any errors. I just cannot get the email to send at this point. I believe that is a thing to work out with my IT dept. 

I am now going to try to get this to work with portal. 

Rickey Fite‌ you can try the following to use with Portal hosted services:

import requests, json, datetime, time, smtplib
from datetime import timedelta

# Disable warnings
requests.packages.urllib3.disable_warnings()

# Variables
username = 'jskinner@ESRI'         # AGOL Username or Portal username
password = 'password123'           # AGOL Password or Portal password

URL = 'https://portal.esri.com/arcgis/rest/services/Hosted/Cities/FeatureServer/0/query'       # Feature Service URL
uniqueID = 'objectid'              # i.e. OBJECTID
dateField = 'created_date'         # Date field to query
hoursValue = 1                     # Number of hours to check when a feature was added

fromEmail = 'manager@esri.com'     # Email sender
toEmail = 'user1@esri.com'         # Email receiver
smtpServer = 'smtp.gis.com'        # SMTP Server Name
portNumber = 25                    # SMTP Server port

AGOL = False                       # True if service is hosted in AGOL, False if service is hosted in Portal
portalServer = 'portal.esri.com'   # Server name portal is installed on (not web adaptor)

# Create empty list for uniqueIDs
oidList = []

# Generate AGOL token
if AGOL == True:
    try:
        print('Generating Token')
        tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
        params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'}
        r = requests.post(tokenURL, data=params, verify=False)
        response = json.loads(r.content)
        token = data['token']
    except:
        token = ''
else:
    # Generate Portal token
    tokenURL = 'https://{0}:7443/arcgis/sharing/rest/generateToken/'.format(portalServer)
    params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'https://{0}'.format(portalServer)}
    r = requests.post(tokenURL, data = params, verify=False)
    response = json.loads(r.content)
    token = response['token']

# Query service and check if created_date time is within the last hour
params = {'f': 'pjson', 'where': "1=1", 'outFields' : '{0}, {1}'.format(uniqueID, dateField), 'returnGeometry' : 'false'}
r = requests.post(URL, data=params, verify=False)
response = json.loads(r.content)
for feat in response['features']:
    createDate = feat['attributes'][dateField]
    createDate = int(str(createDate)[0:-3])
    t = datetime.datetime.now() - timedelta(hours=hoursValue)
    t = time.mktime(t.timetuple())
    if createDate > t:
        oidList.append(feat['attributes'][uniqueID])

print(oidList)

# Email Info
FROM = fromEmail
TO = [toEmail]
SUBJECT = 'New Features Added'
TEXT = "Features with {0}s {1} were added.".format(uniqueID, oidList)

message = """\
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

# If new features exist, send email
if len(oidList) > 0:
    smtpObj = smtplib.SMTP(host=smtpServer, port=portNumber)
    smtpObj.sendmail(FROM, TO, message)
    print("Successfully sent email")
    smtpObj.quit()

Jake Skinner

I got this working Thank you for your help

Jake Skinner

My client wants to just know the number of new records added within the past 24 hrs. 

Is there a way to do that? 

Jake Skinner

My client now wants to just have the number of new records that have been added within the past 24 hrs.

How would I go about doing that? 

Update: I have figured out how to do this but it is not the cleanest code.

Jake,

Yes I can access that URL with no problem at all.

Anthony Whitley‌ try the new code posted above.  The issue is most likely with the outfields parameter.  I had to update this to outFields.

You should just have to change the hoursValue to 24.

I am sorry I do not think I was clear enough. 

I need the email to say

There were  4  sighting(s) added the  Map within the past 24 hrs.

Jake,

Thanks for the reply. I'm testing the other script that was added as well for testing purposes and suspect I might have a Firewall issue. That in ind, the code;

import urllib2, json, urllib, datetime, time, smtplib 
from datetime import timedelta 

name = []
oidList = []
phone = []
email = []
call = []
details = []
mergelist = zip(name, phone, email, call, details)

URL = 'REDACTED' 
params = {'f': 'pjson', 'where': "1=1", 'outfields' : 'OBJECTID, submitted, name, phone, email, call_back, details', 'returnGeometry' : 'false'} 
req = urllib2.Request(URL, urllib.urlencode(params)) 
response = urllib2.urlopen(req) 
data = json.load(response)
for feat in data['features']: 
    createDate = feat['attributes']['submitted'] 
    createDate = int(str(createDate)[0:-3]) 
    t = datetime.datetime.now() - timedelta(hours=1) 
    t = time.mktime(t.timetuple()) 
    if createDate > t: 
        name.append(feat['attributes']['name'])    
        oidList.append(feat['attributes']['OBJECTID'])
        phone.append(feat['attributes']['phone'])
        email.append(feat['attributes']['email'])
        call.append(feat['attributes']['call_back'])
        details.append(feat['attributes']['details'])

def newList():
    mergelist = zip(name, phone, email, call, details)
    for c1,c2,c3 in mergelist:
       return '{} {} {}\n'.format(c1, c2, c3, c4, c5)


FROM = 'fayetteville2019gis@gmail.com' 
TO = ['twhitley@fayetteville-ga.gov'] 

SUBJECT = 'New feature Added' 
TEXT = "Features with" + " " + str(mergelist) + " " + "were added to the GIS layer"

message =""" 'Subject: New feature Added'.format(SUBJECT, TEXT) 
From: %s
To: %s
Subject: %s
 
%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

if len(oidList) > 0:
    smtpObj = smtplib.SMTP_SSL(host='smtp.gmail.com', port=465)
    smtpObj = smtplib.ehlo()
    smtpObj.login('fayetteville2019gis@gmail.com', 'REDACTED')
    smtpObj.sendmail(FROM, TO, message) 
    print "Successfully sent email"
    smtpObj.quit()   
else:
    print "No new data found"

is returning the same error:

Traceback (most recent call last):
  File "C:\Users\twhitley\Downloads\complaint_notification3.py", line 15, in <module>
    response = urllib2.urlopen(req)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 404, in open
    response = self._open(req, data)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 422, in _open
    '_open', req)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "C:\Python27\ArcGIS10.3\lib\urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>

I've tried Google smtp port 465 and 587

Also, tried city smtp settings with the same error message.

I've changed email parameters to include sending from myself to myself with same error message.

Was just hoping for another pair of eyes on the code before I go talk to IT about possible Firewall issues.

I'm attempting to run this script but keep having the same error: KeyError: 'features'. 

I was wondering if you might have a better idea what the issue is? I noticed that a few others in this post had the same error and I attempted to make the changes that you suggested but had no luck.

Thank you so much for your help!

Karlie Kendall‌ can you share the service to an AGOL Group and invite my account, jskinner_CountySandbox?

Karlie Kendall‌ I had a type-o in the code above.  Line 38 in your code should be:

token = response['token']

I am running into an issue where createDate is never greater than t despite there being features created within the time range. I suspect there is a difference between how the CreationDate field values are formatted vs the value of t but I am not sure what that is. 

Code:

import urllib2, json, urllib, datetime, time, smtplib, requests
from datetime import timedelta

oidList = []

#Variables for your ArcGIS online account

username = 'user' #AGOL UserName
password = 'pass' #AGOL password

#Variables

hoursValue = 48
uniqueID = 'OBJECTID'

# Feature Service URL

URL = 'https://services.arcgis.com/URL/FeatureServer/0/query'

# Generate AGOL token
try:
print('Generating Token')
tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'}
req = urllib2.Request(tokenURL, urllib.urlencode(params))
response = urllib2.urlopen(req)
data = json.load(response)
token = data['token']
except:
token = ''

print("Token: " + str(token))

params = {'f': 'pjson', 'where': "1=1", 'outfields' : 'OBJECTID, CreationDate', 'returnGeometry' : 'false', 'token' : token}
req = urllib2.Request(URL, urllib.urlencode(params))
response = urllib2.urlopen(req)
data = json.load(response)


for feat in data['features']:
createDate = feat['attributes']['CreationDate']
createDate = int(str(createDate)[0:-3])

t = datetime.datetime.now() - timedelta(hours=hoursValue)

t = time.mktime(t.timetuple())

if createDate > t:
oidList.append(str(feat['attributes'][uniqueID]))


FROM = 'something@gmail.com'
TO = ['something@something.com']

SUBJECT = 'New Service Requests'

TEXT = "New service requests have been received. Please go to the Service Request Tracking app to review.\n\nhttps://appURLhere"

message =""" 'Subject: New Service Requests Added'.format(SUBJECT, TEXT)
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

gmailPwd = 'gmailpass'

if len(oidList) > 0:
 smtpObj = smtplib.SMTP(host='smtp.gmail.com', port= 587)
 smtpObj.starttls()
 smtpObj.login(FROM, gmailPwd)
 smtpObj.sendmail(FROM, TO, message) 
 print "Successfully sent email" 
 smtpObj.quit()
 
else:
 print "No new data found"

Thank you for the quick response Jake! I’m still getting the same error unfortunately (KeyError: ‘features’).

Karlie Kendall‌ I realized that I needed to incorporate some additional code to handle services that have more than 1,000 features.  Here is the code I was able to get to work with your service:

import requests, json, datetime, time, smtplib, math
from datetime import timedelta

# Disable warnings
requests.packages.urllib3.disable_warnings()

# Variables
username = 'jskinner_CountySandbox'        # AGOL Username
password = '*******'    # AGOL Password

URL = 'https://services.arcgis.com/ouxQl0MYEAGhfutD/arcgis/rest/services/Community_Addresses/FeatureServer/...'       # Feature Service URL
uniqueID = 'OBJECTID'           # i.e. OBJECTID
dateField = 'CreationDate'      # Date field to query
hoursValue = 2                  # Number of hours to check when a feature was added

fromEmail = 'no-reply@esri.com' # Email sender
toEmail = 'analyst@esri.com'   # Email receiver
smtpServer = 'smtp.gis.com'    # SMPT Server Name
portNumber = 25                 # SMTP Server port

# Create empty list for uniqueIDs
oidList = []

# Generate AGOL token
try:
        print('Generating Token')
        tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
        params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'}
        r = requests.post(tokenURL, data=params, verify=False)
        response = json.loads(r.content)
        token = response['token']
except:
        token = ''

# Return largest ObjectID
whereClause = '1=1'
params = {'where': whereClause, 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
r = requests.post(URL, data = params, verify = False)
response = json.loads(r.content)
try:
    response['objectIds'].sort()
except Exception as e:
    print("Error: {0}".format(e))

OIDs = response['objectIds']
count = len(response['objectIds'])
iteration = int(response['objectIds'][-1])
minOID = int(response['objectIds'][0]) - 1
OID = response['objectIdFieldName']

# Query service and check if created_date time is within the last hour
if count < 1000:
    params = {'f': 'pjson', 'where': "1=1", 'outFields' : '{0}, {1}'.format(uniqueID, dateField), 'returnGeometry' : 'false', 'token' : token}
    r = requests.post(URL, data=params, verify=False)
    response = json.loads(r.content)
    for feat in response['features']:
        createDate = feat['attributes'][dateField]
        createDate = int(str(createDate)[0:-3])
        t = datetime.datetime.now() - timedelta(hours=hoursValue)
        t = time.mktime(t.timetuple())
        if createDate > t:
            oidList.append(feat['attributes'][uniqueID])
else:
    y = minOID
    x = minOID + 1000

    ids = response['objectIds']
    newIteration = (math.ceil(iteration/1000.0) * 1000)
    while y < newIteration:
        if x > int(newIteration):
            x = newIteration
        where = OID + '>' + str(y) + ' AND ' + OID + '<=' + str(x)
        print('Querying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
        params = {'f': 'pjson', 'where': where, 'outFields' : '{0}, {1}'.format(uniqueID, dateField), 'returnGeometry' : 'false', 'token' : token}
        r = requests.post(URL, data=params, verify=False)
        response = json.loads(r.content)
        for feat in response['features']:
            createDate = feat['attributes'][dateField]
            createDate = int(str(createDate)[0:-3])
            t = datetime.datetime.now() - timedelta(hours=hoursValue)
            t = time.mktime(t.timetuple())
            if createDate > t:
                oidList.append(feat['attributes'][uniqueID])
        x += 1000
        y += 1000

print(oidList)

# Email Info
FROM = fromEmail
TO = [toEmail]
SUBJECT = 'New Features Added'
TEXT = "Features with {0}s {1} were added.".format(uniqueID, oidList)

message = """\
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

# If new features exist, send email
if len(oidList) > 0:
    smtpObj = smtplib.SMTP(host=smtpServer, port=portNumber)
    smtpObj.sendmail(FROM, TO, message)
    print("Successfully sent email")
    smtpObj.quit()

Ronnie Ragains‌ does your service have more than 1,000 features?  If it does, take a look at the updated code I uploaded today.  Before, it would query with an expression of 1=1, which will only return the first 1,000 records.  I updated the code to iterate through all features, not just the first 1,000.

It is working now!! Thank you so much for your help!

Hi,

I am trying to set up the scrip for working but I've got some issues. I am new to python, so I don't know if I am doing everything correctly.

My Feature Service is stored in GDB Enterprise and publish on ArgGis Onlie via AGS. I have changed the credentials for the account, and the url address, also the information about e-mail and server names, and when I try to run the script tI've received the following issue: 

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>script_agol.py
Generating Token
Error: 'objectIds'
Traceback (most recent call last):
  File "script_agol.py", line 45, in <module>
    OIDs = response['objectIds']
KeyError: 'objectIds'

I have cheched my Feature Service and I have field 'OBJECTID' in it.

Below there is a script which I am trying to set up.

import requests, json, datetime, time, smtplib
from datetime import timedelta

# Disable warnings
requests.packages.urllib3.disable_warnings()

# Variables
username = 'xxx'        # AGOL Username
password = 'xxx'    # AGOL Password

URL = 'https://arcgis.com/arcgis/rest/services/FeatureServer/0'       # Feature Service URL
uniqueID = 'OBJECTID'           # i.e. OBJECTID
dateField = 'created_date'      # Date field to query
hoursValue = 1                  # Number of hours to check when a feature was added

fromEmail = 'sth@gmail.com' # Email sender
toEmail = 'sth@gmail.com'   # Email receiver
smtpServer = 'smtp.gmail.com'    # SMPT Server Name
portNumber = 0                 # SMTP Server port

# Create empty list for uniqueIDs
oidList = []

# Generate AGOL token
try:
        print('Generating Token')
        tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
        params = {'f': 'pjson', 'xxx': username, 'xxx': password, 'referer': 'https://arcgis.com'}
        r = requests.post(tokenURL, data=params, verify=False)
        response = json.loads(r.content)
        token = response['token']
except:
        token = ''

# Return largest ObjectID
whereClause = '1=1'
params = {'where': whereClause, 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
r = requests.post(URL, data = params, verify = False)
response = json.loads(r.content)
try:
    response['objectIds'].sort()
except Exception as e:
    print("Error: {0}".format(e))

OIDs = response['objectIds']
count = len(response['objectIds'])
iteration = int(response['objectIds'][-1])
minOID = int(response['objectIds'][0]) - 1
OID = response['objectIdFieldName']

# Query service and check if created_date time is within the last hour
if count < 1000:
    params = {'f': 'pjson', 'where': "1=1", 'outFields' : '{0}, {1}'.format(uniqueID, dateField), 'returnGeometry' : 'false', 'token' : token}
    r = requests.post(URL, data=params, verify=False)
    response = json.loads(r.content)
    for feat in response['features']:
        createDate = feat['attributes'][dateField]
        createDate = int(str(createDate)[0:-3])
        t = datetime.datetime.now() - timedelta(hours=hoursValue)
        t = time.mktime(t.timetuple())
        if createDate > t:
            oidList.append(feat['attributes'][uniqueID])
else:
    y = minOID
    x = minOID + 1000

    ids = response['objectIds']
    newIteration = (math.ceil(iteration/1000.0) * 1000)
    while y < newIteration:
        if x > int(newIteration):
            x = newIteration
        where = OID + '>' + str(y) + ' AND ' + OID + '<=' + str(x)
        print('Querying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
        params = {'f': 'pjson', 'where': where, 'outFields' : '{0}, {1}'.format(uniqueID, dateField), 'returnGeometry' : 'false', 'token' : token}
        r = requests.post(URL, data=params, verify=False)
        response = json.loads(r.content)
        for feat in response['features']:
            createDate = feat['attributes'][dateField]
            createDate = int(str(createDate)[0:-3])
            t = datetime.datetime.now() - timedelta(hours=hoursValue)
            t = time.mktime(t.timetuple())
            if createDate > t:
                oidList.append(feat['attributes'][uniqueID])
        x += 1000
        y += 1000

print(oidList)

# Email Info
FROM = fromEmail
TO = [toEmail]
SUBJECT = 'New Features Added'
TEXT = "Features with {0}s {1} were added.".format(uniqueID, oidList)

message = """\
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

# If new features exist, send email
if len(oidList) > 0:
    smtpObj = smtplib.SMTP(host=smtpServer, port=portNumber)
    smtpObj.sendmail(FROM, TO, message)
    print("Successfully sent email")
    smtpObj.quit()

Can someone advice what should I do to set it for run?

Thanks in advance,

Monika

Monika Samorajska‌ you will need to add /query at the end of your URL.  Ex:

A couple other things:

1.  There is a feature where the create_date is NULL.  This will cause the tool to error.  You will need to a line of code to skip this record (i.e. if createDate is not None).  See linee 58 below:

2.  Also, with this being an ArcGIS Server feature service, you should apply security at the ArcGIS Server level.  Currently, this service can be edited by anyone who has access to the URL.  After you secure the service, you will need to specify an ArcGIS Server username/password, and you will need to update the code to generate the token:

# Generate AGOL token
try:
  tokenURL = 'https://arcgis.egis.fr/arcgis/tokens/generateToken'
  params = {'f': 'pjson', 'username': username, 'password': password, 'client': 'requestip'}

  r = requests.post(tokenURL, data = params, verify=False)
  response = json.loads(r.content)
  token = response['token']
except:
  token = ''

Hi Jake,

Thanks Jack for very quick respond. I really appreciate that.

I have made the changes which you pointed out, but I've found another issue. This time the error message shows me the following issue:

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>script_agol_v2.py
  File "script_agol_v2.py", line 58
    if createDate is not None:
                             ^
TabError: inconsistent use of tabs and spaces in indentation

Below how I have changed the script:

Monika

Monika Samorajska‌ you will just need to tab over everything below if createDate is not None.  Ex:

# Query service and check if created_date time is within the last hour
if count < 1000:
    params = {'f': 'pjson', 'where': "1=1", 'outFields' : '{0}, {1}'.format(uniqueID, dateField), 'returnGeometry' : 'false', 'token' : token}
    r = requests.post(URL, data=params, verify=False)
    response = json.loads(r.content)
    for feat in response['features']:
        createDate = feat['attributes'][dateField]
        if createDate is not None:
            createDate = int(str(createDate)[0:-3])
            t = datetime.datetime.now() - timedelta(hours=hoursValue)
            t = time.mktime(t.timetuple())
            if createDate > t:
                oidList.append(feat['attributes'][uniqueID])

Hi Jake,

thanks for every advice. I was managed to copy your part of the scrit and as a result a received information like below:

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>script_agol.py
Generating Token
[]

When I edit the feature service and run script again I receive below information, but in the command prompt but still I don't receive an e-mail with the informaion that FC has been changed.

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>script_agol.py
Generating Token
Traceback (most recent call last):
  File "script_agol.py", line 63, in <module>
    oidList.append(feat['attributes'][uniqueID])
KeyError: 'OBJECTID'

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>

Can yo advice what should I do or do you know what's wrong I am doing?

Thanks in advance,

Monika

Monika Samorajska‌ on line 12, change OBJECTID to objectid:

uniqueID = 'objectid'   

Hey Jake,

much appriaciated for your help. Script works right now, it recognize that the new feature was added by, because I have some server connection issues it doesn't send me an email. I will discuss this issue with my IT department.

I was just wondering, is it possibile to change the script, especially line 13 where is dateField = "created_date" to "last_edited_date" so the script can show all the modification, not only new features in the FC. Or is it more complicated than I think?

Wishes,

Monika

Monika Samorajska‌ yes, you can change line 13 to "last_edited_date" and you will receive notifications based on this field.

Hey Jack,

thank you very much for your help!.

I've got one more issue, and I was hoping you will help me understand what is the erros which I received from the script. Please see the  information below:

sock.connect(sa)
OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions

I am thinking that this issue can be on my firewall and no permission for sth, but before I will peak with the IT department, I would like to understand what exaclty I should tell them to fix this error. 

Wishes,

Monika

Not quite sure what is causing this issue, but it appears to be related to when you're sending the e-mail.  May want to follow up with IT.

Hey,

thank you so much for your help.

I am really appreciated for that.

Wishes,

Monika

Hi Jake,

I tried to run the script with my personal account which is on gmail SMTP, but unfortunatelly I am having another issue. This time it is saying :

 raise SMTPSenderRefused(code, resp, from_addr)
smtplib.SMTPSenderRefused: (530, b'5.7.0 Must issue a STARTTLS command first. p9sm20460115wrs.55 - gsmtp', 'msamorajska@gmail.com')

I was searching for the solution on the Internet, and I found that I should provide credentials for my gmail account. I should say, that credentials for AGOL account and for gmail accout are different, and can you advice where should I add these information?

Or do you have another explanation of my issue?

Thanks in advance for your help!

Monika

Monika Samorajska‌ take a look at the following article:

How to Send an Email With Python and smtplib? (in 5 lines) - Afternerd 

You most likely need to enable less secure apps for your gmail account.  Then you can use the following code:

server = smtplib.SMTP_SSL('smtp.gmail.com', 465)
server.login("user@gmail.com", "*******")

FROM = 'user@gmail.com'
TO = ['to@esri.com']
SUBJECT = 'New Features Added'
TEXT = "Features added."

message = """\
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

server.sendmail(FROM, TO, message)
server.quit()

Thanks, now it is working with my gmail account.

Monika

Hi,

I've got another question. This script is looking for the created date and according to this field it is detecting the features which had been changed. Can you advise how to modify the script in the way that the script will look for the changes in related tables and according to the field detected the features with changes.

Monika

You should just need to update the URL variable to the REST Service URL of the related table, and the dateField variable to the date field in the related table.

Hi,

thanks for the advice. I also would like to ask about how to add the information to the e-mail about the changes. I mean, when the script detects some changes, in the process when token is generating, I can see the number of the object which has been changed. Can you please advice how can I add this nformation to the e-mail which is genetared atomatically?

Thanks again for any help.That is really helpfull for me.

Wishes,

Monika

Hi Jake Skinner‌ i run this code in my python interpreter with this error.Need help please...

Generating Token
Querying features with ObjectIDs from 0 to 1000
Traceback (most recent call last):
  File "C:\Users\SIG02-PC\Desktop\MailSending.py", line 79, in <module>
    for feat in response['features']:
KeyError: 'features'
>>> 

The code

import requests, json, datetime, time, smtplib
from datetime import timedelta
import math

# Disable warnings
requests.packages.urllib3.disable_warnings()

# Variables
username = '********'        # AGOL Username
password = '************'    # AGOL Password

uniqueID = 'objectid'           # i.e. OBJECTID
dateField = 'Created_Date'      # Date field to query
hoursValue = 1                  # Number of hours to check when a feature was added

fromEmail = 'anselme.s@hitech-services.com' # Email sender
toEmail = 'anselme.s@hitech-services.com'   # Email receiver
smtpServer = 'mail.hitech-services.com'    # SMPT Server Name
portNumber = 587                 # SMTP Server port

# Create empty list for uniqueIDs
oidList = []

# Generate AGOL token
try:
        print('Generating Token')
        params = {'f''pjson''username': username, 'password': password, 'referer''http://www.arcgis.com'}
        r = requests.post(tokenURL, data=params, verify=False)
        response = json.loads(r.content)
        token = response['token']
except:
        token = ''

# Return largest ObjectID
whereClause = '1=1'
params = {'where': whereClause, 'returnIdsOnly''true''token': token, 'f''json'}
= requests.post(URL, data = params, verify = False)
response = json.loads(r.content)
try:
    response['objectIds'].sort()
except Exception as e:
    print("Error: {0}".format(e))

OIDs = response['objectIds']
count = len(response['objectIds'])
iteration = int(response['objectIds'][-1])
minOID = int(response['objectIds'][0]) - 1
OID = response['objectIdFieldName']

# Query service and check if created_date time is within the last hour
if count < 1000:
    params = {'f''pjson''where'"1=1"'outFields' : '{0}{1}'.format(uniqueID, dateField), 'returnGeometry' : 'false''token' : token}
    r = requests.post(URL, data=params, verify=False)
    response = json.loads(r.content)
    for feat in response['features']:
        createDate = feat['attributes'][dateField]
        createDate = int(str(createDate)[0:-3])
        t = datetime.datetime.now() - timedelta(hours=hoursValue)
        t = time.mktime(t.timetuple())
        if createDate > t:
            oidList.append(feat['attributes'][uniqueID])
else:
    y = minOID
    x = minOID + 1000

    ids = response['objectIds']
    newIteration = (math.ceil(iteration/1000.0* 1000)
    while y < newIteration:
        if x > int(newIteration):
            x = newIteration
        where = OID + '>' + str(y) + ' AND ' + OID + '<=' + str(x)
        print('Querying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
        params = {'f''pjson''where': where, 'outFields' : '{0}{1}'.format(uniqueID, dateField), 'returnGeometry' : 'false''token' : token}
        r = requests.post(URL, data=params, verify=False)
        response = json.loads(r.content)
        for feat in response['features']:
            createDate = feat['attributes'][dateField]
            createDate = int(str(createDate)[0:-3])
            t = datetime.datetime.now() - timedelta(hours=hoursValue)
            t = time.mktime(t.timetuple())
            if createDate > t:
                oidList.append(feat['attributes'][uniqueID])
        x += 1000
        y += 1000

print(oidList)

# Email Info
FROM = fromEmail
TO = [toEmail]
SUBJECT = 'New Features Added'
TEXT = "Features with {0}{1} were added.".format(uniqueID, oidList)

message = """\
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

# If new features exist, send email
if len(oidList) > 0:
    smtpObj = smtplib.SMTP(host=smtpServer, port=portNumber)
    smtpObj.sendmail(FROM, TO, message)
    print("Successfully sent email")
    smtpObj.quit()

Hi Jake Skinner‌,

Need help please,

Anselme,

GIS Developer

Dambé SANWOGOU‌ can you share the service to an ArcGIS Online group and invite my account (jskinner_CountySandbox)?

Version history
Last update:
‎08-02-2021 10:30 AM
Updated by:
Contributors