A simple e-mail notification system for Survey123 for ArcGIS v2

17474
101
01-22-2018 03:55 PM
Esri Frequent Contributor
17 101 17.5K

Back in November 2016, I published a blog post describing how to automate e-mail notifications against a Survey123 project. The idea was simple: A Python script that would trigger an email  every time a new survey is submitted. By scheduling the script to run unattended at certain time intervals, one could easily put together a simple e-mail notification system.

I want now to revisit this topic and share a new Python script, which takes care of some limitations of the first version. Specifically, the new script:

  1. Works against ArcGIS Enterprise in addition to ArcGIS Online
  2. Can be configured to either send one email for every newly added survey, or alternatively send one email for all new records added since the script was last launched.

While the Python script can be used as is, I really share it as a starting point from which you can write the exact logic that matches your needs. You may for example tweak the script to:

  • Format the e-mail notification following specific branding guidelines.
  • Extend the notification logic so the recipient of the email varies depending on data submitted.

Below are step by step instructions so you can configure this Python script with your own survey.

Preparing your computer to run the Python script

Technically, you can host the script in any computer that is able to run Python 3.x, however, you will want to select the host carefully. You will want  the script to run regularly and unattended, so you want to pick a computer that will reliably running 24x7 and constantly connected to your network.

In addition to Python 3.x, the script requires the 'requests' Python module installed.  If your computer already has ArcGIS Pro installed, you already have Python 3.x and the 'requests' module installed. However, ArcGIS Pro is not required to run this script.

Preparing your feature service

The Python script works against your survey's feature service. The script requires that your feature service has Editor Tracking enabled. By default, all feature services created by Survey123 have Editor Tracking enabled, so there is really nothing that you need to do extra. If  your survey is configured to work against a service from ArcGIS Server, then you will want to make sure  Editor Tracking is enabled, and if not, enable it manually.

If you do not enable Editor Tracking, the  script will generate an error  text file indicating that Editor Tracking needs to be enabled and will stop execution.

Downloading and configuring the Python script

Now you can download and configure the script as follows:

  1. Download the Detect Changes and Notify Python script v2 and its associated init.json file and save them together within a folder in your computer.
  2. Edit and save the init.json file. At the very least, you will need to change the properties highlighted  in blue below. 

{
   "email":{
     "recipients":["JohnSmith@acme.com"],
      "from":"acme@acme.com",
      "subject":"New features has been added to your service",
      "text":"Hi Team,\n\nYou are receiving this e-mail because features have been added in:\n",
  "server":["smtp.acme.com", "", "", ""],
"onemailflag":1
   },
   "service":{
      "portalURL":"https://www.arcgis.com",
"fsURL":"https://services2.arcgis.com/fJJEXNgxjn0dpNsi/arcgis/rest/services/service_20dc5e3eab3f4efab8fde88f6...",
      "fsLayerNum":0,
  "serviceuser":"yourusername",
"servicepw":"yourpassword",
      "fieldstoreport":["*"],
      "viewerMapLevel":19
   },
   "filenames":{
      "lasteditfile":"lastedit.json"
   }
}

  • recipients: This is the e-mail that will be receiving notifications. You can create an email alias if you want several people to receive the email.
  • from: This is the e-mail of the person/organization that is sending the e-mail. You can also set it to a DoNotReply email address.
  • server: This setting refers to your email server connection properties which are defined as a comma separated list of strings. 
    • The first parameter is the host of your email server which can be specified by hostname or IP address. It will look something like smtp.yourCompany.com for example or smtp.gmail.com
    • The second parameter is optional and defines the port where the SMTP server is listening. Common ports are 25 and 587 but it could really be any other port depending on how your mail server is configured.
    • The third and fourth parameters are also optional and are used to set a user and password to access your email server.
To  properly get the email  server connection properties, you will want to contact your IT department and describe what you are trying to  achieve as many email servers are configured with strict security  policies that will prevent the script from successfully connecting and  using your corporate e-mail server.  Folks in your IT department should  know how to give you the right hostname and port for  your email server.  Handing over the source code of the Python script may be of help too.

If you want to configure this script using your Gmail,  You will need to configure your Gmail account with 2-step verification, and then setup a Gmail App password.   Once you have done that, use your complete Gmail e-mail address as the  user (third parameter) and the App password for the fourth parameter.   When using Gmail you do not need to specify a port number, so simply type "" in the second parameter.
    • onemailflag: If you leave this parameter as 1, then an e-mail will be sent summarizing all records added since the Python script executed for the last time. If you switch the value to 0, then the script will trigger one e-mail for every record added.
    • portalURL: You only need to change this setting if working against ArcGIS Enteprise. You will want here the full https location of your portal including the Web Adaptor.
    • fsURL: This is the URL of your survey's feature service, which could be running in ArcGIS Online or ArcGIS Enterprise.
    • fsLayerNum: This is the index of the layer in the feature service that you want to monitor. If you want to check for changes in the main layer of your survey, leave it to 0. If you want to check for changes in the layer of a repeat, change the index accordingly.
    • serviceuser and servicepw: The ArcGIS credentials of a user with access to the feature service.

After saving changes to the init.json file, it is time to give the script  a quick test:

  1. Run the script once so it can capture the current state of your feature service.
  2. Submit one record to your feature service and run the script again.

If any errors occur during execution, an error text file will be created in the same folder where you saved the Python script.

Scheduling the script

The Windows operating system includes a simple utility called Task Scheduler. It is quite easy to setup. Once you have determined  when the script will be triggered for the first  time, you can repeat its  execution at regular intervals. For example, every 5 minutes. Obviously, the computer where  you setup the task will need to be running all the time, although you can configure the task to run regardless of who is logged in.

The configuration of tasks in the scheduler is pretty much self-explanatory, but here are some specific instructions that can save you some back and forth:

  1. General: Check the option to run with highest privileges and set the task to run even if you are not logged-in.
  2. Trigger: If you want to quickly test your task, you can simply select your task in the gallery and then hit Run in the Selected Item panel on the right.  When configuring the task for real, I suggest you select the startup trigger and that you also configure the task to run indefinitely every five or ten minutes or so.
  3. Actions: You will need to be particularly careful with this one. The Program/Script setting needs to point to the Python executable (Python.exe).
    • If using your own copy of Python, refer to the installation directory of Python where you will find Python.exe
    • If using Python from ArcGIS Pro, it will be under the Pro installation directory. For example: "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe"

You also need to indicate the location of your DetectChanges.py Python script as an argument. Do not forget to include the .py extension in the path. If the path to your DetectChanges.py file includes spaces, then you need to enclose the path with quotes. Lastly, set the 'Start in' property so it points to the  directory in which you are storing the DetectChanges.py file.

Beyond the basics

The technique described in this blog post is a bit rudimentary, but it  may do its job reasonably in some simple scenarios. You can manipulate the Python script to properly format the email message to be sent, and also to apply some logic to determine if a n email needs to be sent, and to who.  Since you are in 'Python land' already, you can get creative and import the arcpy.mapping module to do all sort of sophisticated things in the script. You can for example take the incoming feature added into your feature service and do a straight (buffer) or network distance (closest facility) search to determine who will be notified.

Differences between the original script and this one

There are a handful of differences between the original Python script published in November 2017 and this one.  For the most part, they are the same, but here are the key changes:

  • The most important difference is how changes in the feature layer are found. In the November 201 script the script exercises a new API introduced in ArcGIS Online to detect changes. This new API only works in ArcGIS Online (not in Enteprise) and it had to be enabled manually on the survey's feature service.  The latest version of the script, described in this  post, uses Editor Tracking. Editor Tracking is enabled by default in every feature service created by Survey123 for ArcGIS. Editor Tracking is compatible with ArcGIS Online feature services and also with ArcGIS Server feature services back to 10.1.
  • One email per feature or one per execution: The other  key difference is that in our November 201 version, it was not possible to easily configure the script to aggregate all changes in a single email. With this version, you can use the 'onemailflag' parameter in the init file to decide if you want to send one email per added record, or one email per execution of the script.

If you are already using the November 201 version of the script and want to switch to this one, I recommend that you copy the Python script and the new init file  into a separate folder. The init file is different, so pay attention to the new parameters introduced.

101 Comments
New Contributor III

Clicked the link  'Download the Detect Changes and Notify Python script v2 and its associated init.json' and it took me to a login page where my credentials don't work. Is there another way I can download these files to test them or can the content be shared? The previous blog post allowed me to download the files no problem.

Esri Frequent Contributor

Please try again. Item is now shared

New Contributor III

Thank you sir!

New Contributor

Please I get this error when I run the scripts

" ConnectionError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/generateToken (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x000000000A80F8D0>: Failed to establish a new connection: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',)) "

I would appreciate your advise

Esri Esteemed Contributor

Hi Patrick,

I'm not sure it's related, but the getToken() function on line 63 looks to not be closed - you commented out some of the later parameters but didn't supply a closing )

New Contributor III

Script is only sending to the first recipient. Is there a proper way to format "recipients": to allow for multiple email addresses?

Esri Esteemed Contributor

Hi Glen, 

How did you specify the addresses?  According to 20.12. smtplib — SMTP protocol client — Python 2.7.14 documentation , they should be a list of addresses:

["one@one.org","two@two.org","three@three.org"]

New Contributor III

Thanks James. I was misplacing the quotes around the whole list of recipients.

Occasional Contributor III

I followed all the steps mentioned in this article using my Gmail account.

But, I kept getting this error message.

Unable to generate token.
Invalid username or password.

(in getToken)

Does anyone know what this could be due to?

Thanks,

Gee

Esri Esteemed Contributor

Hi Geethaka,

This indicates that the script does not have a valid login for you into your ArcGIS Organization (Online/Enterprise)

Occasional Contributor II

I will definitely use this to improve my Survey123 security application - will keep you posted!

New Contributor II

Excellent script, thanks for keeping this updated.

I should mention that, despite what the guide said on configuring the script to work with Gmail, I did have to include a port for the 2nd parameter for "server", specifically it was port 587. 

Hopefully that helps others moving forward,

Tom

New Contributor

Worked like a champ and is extremely helpful in our organization!!

Thank you so much for posting!!

-Marla 🙂

by
Occasional Contributor III

Hello, ichivite-esristaff    

I am having issue with this. Am I supposed to input info in "DetectEdits.py" first then run the "initDetectEdits.json"? I am not sure if I am understanding correctly. Please advise. Thank you.

New Contributor

This is very helpful, and it worked well after a bit of trial and error on my part.  Thank you very much.  Has anyone customized the script to add attribute information to the subject line?

New Contributor III

It's important to note that email.message.Message() in Python 2.7 requires the recipients as a single comma-delimited string, while smtp.sendmail requires a list of individually quoted recipient addresses:

eMailFrom = "me@myco.com"

eMailTo = ["Joe@smtp1.com", "Jane@smtp2.com"]

if type(eMailTo) is list:
   toStr = ', '.join(eMailTo)
else:
    toStr = eMailTo

(A single recipient can be either a string or a list of length 1 for sendmail).

Example:

msg = Message()   #note: different from EmailMessage() in Python 3

msg['From'] = eMailFrom

msg['To'] = toStr

msg['Subject'] = 'Message subject'

msg.set_payload(messagetext)   #note: different from set_content in Python 3

.

.

#send the email via SMTP server ss1:

ss1.sendmail(eMailFrom, eMailTo, msg.as_string())

New Contributor III

This looks very useful. I've managed to get the original script version adapted to Python 2.7 and working fine, but the updated version falls over for me at the GetMaxDate function with the message written to the error log:

'outStatistics' parameter is invalid

I get the same error if I try using the REST API interactively to query my survey layer, which is a stock-standard Survey123 layer with the following capabilities:

I'm not that familiar with the REST interface, but I basically added in the same parameters the script passes to the requests url.

Any suggestions, or does anyone else have this problem?

ismael chivite‌

#james tedrick

New Contributor

I was having the same and saw that you tried passing the same parameters to the request url.

I took a look the the URL more carefully and noticed and unwanted character in my URL:

https://services2.arcgis.com/fJJEXNgxjn0dpNsi/arcgis/rest/services/service_20dc5e3eab3f4efab8fde88f6.../0/query?f=pjson&where=1=1&outFields=*&outStatistics=[{'onStatisticField': u'CreationDate', 'statisticType': 'max', 'outStatisticFieldName': 'lasteditdate'}]&token

This u is what was causing my problem. In order to fix this I went back getMaxDate() function and changed statfield to a string:

def getMaxDate(url, statfield, outfield, token😞
    stats = [{"onStatisticField": str(statfield), "statisticType": "max", "outStatisticFieldName": outfield}]

Hope this helps.

New Contributor III

I am a Python newbie and just implemented this in our organization for a Survey123 project.  Thank you so much for making this code so easy to follow and available to everyone!  It's a game changer for those responsible for monitoring intermittent, crowd-sourced data.

I am having one problem with the first link in the email that is sent out.  It looks like:

You are receiving this e-mail because features have been added in:

                https://services.arcgis.com/xxxxxxxxxxxxxxxxxx/arcgis/rest/services/survey123_f9325b9628d7403f8ebdf0...

I get the following error message.

The link needs a token at the end to open correctly.  It should look something like this:

https://services.arcgis.com/xxxxxxxxxxxxxxxxxx/arcgis/rest/services/survey123_f9325b9628d7403f8ebdf0ac9150893e/FeatureServer?token=NTelwbKPuvneQXB...

Any thoughts on how to generate the necessary token at the end of the link in the email?

Thanks,

Becky

New Contributor III

First you edit "initDetectEdits.json". Then you run "DetectEdits.py".

New Contributor III

Thanks, Anthony, that did the trick!

New Contributor

I edited the init.json file, and I believe I followed the directions correctly, but then when I run DetectEdits, I get this error message:

Traceback (most recent call last):
File "C:/Users/intern/Desktop/FLEET MANAGEMENT/DetectEdits.py", line 19, in <module>
from email.message import EmailMessage
ImportError: cannot import name EmailMessage

I can't figure out what 'EmailMessage' is supposed to be, or where it's supposed to be pulling from in order to import it. My understanding was that as long as changes were made in the json file, the python script would run as is. Did I misunderstand, or is there a different error occurring?

Thank you!

Esri Esteemed Contributor

Hi Justin,

The most common issue that would generate this message is using the wrong version of Python (Python 3.x, the version that comes with ArcGIS Pro) - can you check which version you are using?

New Contributor

I was using Python 3.6, which I downloaded outside of ArcGIS Pro, but now that I'm running it in Python 3.5.3 that is packaged with Pro, I'm getting a different error of:

SyntaxError: <unicode error> 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXX escape

I am very much beginner level when it comes to Python, and it seemed simple enough to change the json file to what I wanted, it's the implementation now that's tripping me up.

Thanks James for the quick reply!

Esri Esteemed Contributor

Hi Justin,

it sounds like there's a string (likely a directory path) that begins with a \U somewhere in your configuration, normally Python will try to interpret the next set of characters as a number specifying a unicode character. The normal way to resolve is to use a "raw string" br placing an r in front of the string; see python - Why do I get a SyntaxError for a Unicode escape in my file path? - Stack Overflow 

New Contributor

Hi,

I have tried writing my server in the Init.json file in a number of ways including port and tried with/without login details but consistently get this error:

An email error occurred (in sendMail) while attempting to connect to: smtp.xxxxxxxxxxxxxx.com.au
Please ensure your Mail Server Host and port are set correctly in your Init.json file

Do you have any advice? 

Thanks,

Loren. 

New Contributor III

 I'm a Python newbie and just implemented this in our organization for a Survey123 project.

I have a problem with de proxy.

In module requests.adapters set de proxies that this:

proxies = {
'http': 'http://username:password@server:port/',
'https': 'https://username:password@server:port/',
}

When exuecuting DetectEdits.py results:

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.fcoromero.maps.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/generateToken (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x00000000045D6B38>: Failed to establish a new connection: [Errno 11004] getaddrinfo failed',))

I would appreciate any help.

Thanks.

Occasional Contributor III

Hi all,

I have created a Survey123 workflow with multiple feature layer views.

e.g. Once a survey is reviewed, it'll copy into a different feature layer view for APPROVAL (features defined as REVIEWED = YES).

I managed to get this email workflow to work on my parent feature layer. Thanks ichivite-esristaffJTedrick-esristaff

But, I cannot get it to work on any of the views.

I'm assuming this is because - the records copied into views aren't new records (but records with certain conditions).

But, I would really appreciate it if anyone has any suggestions to get this to work.

Thanks in advance.

Gee

Esri Frequent Contributor

Hi.  This is more likely because the view has a filter or because the view has not been refreshed to reflect new fields added into the parent layer.  Features added to the parent layer will automatically show in the view, unless a filter has been set in the view.  If you add new fields in the parent layer, you will want to go into the definition of the layer and make sure all fields are visible.

Occasional Contributor III

Hi ichivite-esristaff

Thanks so much for your helpful comments.

I'm really sorry for not wording my question properly.

I forgot to mention that I can get the python email script to work on views too, as long as I haven't run the python script on the parent layer beforehand.

  • The issue only occurs, when a new record is added to the parent layer, and I run the python script on the parent layer.
  • Afterwards I change one of the fields in this record (e.g. change reviewed to "YES").
  • It'll then copy this record into a new feature layer view (features defined as REVIEWED = YES).
  • When I try to run the python script on this feature layer view, it wouldn't work.

I think this is due to me previously running the script on that same record when it was in the parent feature layer.

Any thoughts on how I could get around this?

Thanks,

Gee

by
Occasional Contributor III

Hello,

What is the different with this link documentation? Send email notifications - Crowdsource Manager | ArcGIS for State Government 

Please advise.

Thanks!

Esri Esteemed Contributor

Hi Alex,

There aren't many differences between the two; they are essentially two patterns to accomplish the same result.

New Contributor

Hi James,

I am relatively new to python, but I am wondering if there is a way to choose who to send an email to based on a select_one answer.

I have a survey for general field observations and I would like to route the data collected to the person responsible for follow-up. For example, someone finds a blown out culvert and from a select_one question, selects the resource impacted as "Transportation", someone else finds a poached deer and selects the resource impacted as "Wildlife".  Can I send these entries to two different people/supervisors?

Esri Esteemed Contributor

Hi Shannon,

The script as written does not have this capability; revising it to would take a bit of Python programming to look at hte data and determine the correct recipient.  That being said, there is another possibility in development that might address this need without Python coding - see https://community.esri.com/groups/survey123/blog/2018/03/05/sneak-peek-survey123-and-webhooks-integr... 

New Contributor III

Fantastic script! Thank you very much!. This was a request from our highway department, and even as a newbie to python, I was able to get this going ---- it also forced me to upgrade form python 2.7, when the script was giving me errors. Working like a charm after upgrading to 3.6.2 --- I very much appreciate your sharing. Now, here's hoping the move from ArcMap to ArcPro goes just as smoothly.... 

by
New Contributor II

Hello all,

I am attempting to use this script for a survey that includes multiple repeats, generating 2 related tables in my feature layer. I am wondering if it is possible for this script to be modified so that if a user submits a survey that utilizes both repeats, there is only one email that is sent that details the changes in all three layers. To my knowledge, the only support for this problem right now is to create three separate scripts by modifying the fsLayerNum in the init file, which would generate separate emails for each layer.

Thanks for your help!  

James

Esri Esteemed Contributor

Hi James,

It is possible to modify the script.  The key to accessing related records would be to use the Query Related Records (Feature Service)—ArcGIS REST API: Services Directory | ArcGIS for Developers REST function to collect the related records in the feature-by-feature processing part (the for loop that begins at line 177; probably after the msg writing of the main feature that ends at line 215.  You would need to:

- use the queryRelatedRecords function to get the records

- for each record, process the attributes/geometry (essentially the same code from line 198-215)

by
New Contributor II

Hi James,

Thanks for your help. I am still having some trouble getting this to work. I used the related records API and a get request in an attempt to pull the records from the related table, but the page always locks me out, requiring a token. Do I need to generate a new token for each related table, or just utilize the token that is already being created by the script? How would I go about doing either of these options?

Thanks again,

James

Esri Esteemed Contributor

Hi James,

You should be able to utilize the same token - every request should have

token: <token value>

f: json

in the request parameters to ensure security and data format are set properly.

New Contributor II

Hello,

I am receiving the following error when running the script. 

Traceback (most recent call last):
File "E:\Damage Assessment Python\DetectEdits.py", line 340, in <module>
main(configfilename)
File "E:\Damage Assessment Python\DetectEdits.py", line 81, in main
maxeditdate = getMaxDate(urlLyr, editfieldsDict['creationDateField'], 'lasteditdate', servicetoken) / 1000. + 0.001
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

Any ideas?

New Contributor

Wonderful work. Ismael. Thank you very much for share!

MVP Regular Contributor

Great script!

How would I modify the script to only look at a particular field for changes? I'm looking to watch my status field change for a request.

Thanks

Occasional Contributor II

Good morning.

I've got a particular request from our staff, and was wondering if we could leverage this script to also send a notification to the user that submits a web survey. This email would include the report generated using the Report Template feature from Survey123 website for the user's records. Not sure if there is actually a way to auto-generate this, but definitely a feature of great value.

FYI, the Survey will be shared to everyone, so we our idea is to add a question where the user will choose whether or not he would like to receive a printed version of the Survey, and provide and email address accordingly.

Thanks.

Javier

Esri Esteemed Contributor

Hi Javier,

The python script could be adapted as you describe, though it would be a significant change (especially the feature report inclusion).  This is a workflow we would suggest webhooks as a more appropriate solution- see Sneak Peek: Survey123 and Webhooks (Integromat).

New Contributor III

James,

Thanks for taking the time; I'll verify script version and retry.

New Contributor III

James,

Wanted to thank you for pointing me to the more current script. It did resolve my problem and am now troubleshooting the sendmail error.

Again, thank you for such a fantastic script.

Anyone else out there having sendmail errors? My specific error: " An email error occurred (in sendMail) while attempting to connect to: ptc-mail-2016 Please ensure your Mail Server Host and port are set correctly in your Init.json file " I've been assured from my IT folks that the SMTP is correct.

Esri Esteemed Contributor

I'm not sure if it is required, but I would suggest getting the fully qualified domain name (FQDN) - it should be ptc-mail-2016.<domain>.<top level domain>

Additionally, you might need to double check the port settings.

New Contributor III

James,

Thank you. I'll have a look at that.

New Contributor II

In case you have an error with token and you are sure of your username and password. make sure your username is in the same case ( upper case, lowercase, one upper case letter or other ) as in your AGOL profile.

New Contributor III

Okay so on to my last issue: Email setting working (had to go with gmail in the end) but trying to get the detectedits.py to run in Task Scheduler (even using a bat file) always results in "Configuration file (initDetectEdits.json) does not exist...returning"

I've gone so far as copying the entire Detects file and folders within the python 3 environment folder for testing and still the same issue.

simple bat file...

The error...

However, if I run the bat file directly (clicking within the folder)..runs as expected...

I've tested without a bat file use and calling on the detectedits.py directly from Task Scheduler with the same error.

Any thoughts anyone?