A simple e-mail notification system for Survey123 for ArcGIS v2
Back in November 2016, I published a blog post describing how to automate e-mail notifications against a Survey123 project. The idea was simple: A Python script that would trigger an email every time a new survey is submitted. By scheduling the script to run unattended at certain time intervals, one could easily put together a simple e-mail notification system.
I want now to revisit this topic and share a new Python script, which takes care of some limitations of the first version. Specifically, the new script:
- Works against ArcGIS Enterprise in addition to ArcGIS Online
- Can be configured to either send one email for every newly added survey, or alternatively send one email for all new records added since the script was last launched.
While the Python script can be used as is, I really share it as a starting point from which you can write the exact logic that matches your needs. You may for example tweak the script to:
- Format the e-mail notification following specific branding guidelines.
- Extend the notification logic so the recipient of the email varies depending on data submitted.
Below are step by step instructions so you can configure this Python script with your own survey.
Preparing your computer to run the Python script
Technically, you can host the script in any computer that is able to run Python 3.x, however, you will want to select the host carefully. You will want the script to run regularly and unattended, so you want to pick a computer that will reliably running 24x7 and constantly connected to your network.
In addition to Python 3.x, the script requires the 'requests' Python module installed. If your computer already has ArcGIS Pro installed, you already have Python 3.x and the 'requests' module installed. However, ArcGIS Pro is not required to run this script.
Preparing your feature service
The Python script works against your survey's feature service. The script requires that your feature service has Editor Tracking enabled. By default, all feature services created by Survey123 have Editor Tracking enabled, so there is really nothing that you need to do extra. If your survey is configured to work against a service from ArcGIS Server, then you will want to make sure Editor Tracking is enabled, and if not, enable it manually.
If you do not enable Editor Tracking, the script will generate an error text file indicating that Editor Tracking needs to be enabled and will stop execution.
Downloading and configuring the Python script
Now you can download and configure the script as follows:
- Download the Detect Changes and Notify Python script v2 and its associated init.json file and save them together within a folder in your computer.
- Edit and save the init.json file. At the very least, you will need to change the properties highlighted in blue below.
{
"email":{
"recipients":["JohnSmith@acme.com"],
"from":"acme@acme.com",
"subject":"New features has been added to your service",
"text":"Hi Team,\n\nYou are receiving this e-mail because features have been added in:\n",
"server":["smtp.acme.com", "", "", ""],
"onemailflag":1
},
"service":{
"portalURL":"https://www.arcgis.com",
"fsURL":"https://services2.arcgis.com/fJJEXNgxjn0dpNsi/arcgis/rest/services/service_20dc5e3eab3f4efab8fde88f6...",
"fsLayerNum":0,
"serviceuser":"yourusername",
"servicepw":"yourpassword",
"fieldstoreport":["*"],
"viewerMapLevel":19
},
"filenames":{
"lasteditfile":"lastedit.json"
}
}
- recipients: This is the e-mail that will be receiving notifications. You can create an email alias if you want several people to receive the email.
- from: This is the e-mail of the person/organization that is sending the e-mail. You can also set it to a DoNotReply email address.
- server: This setting refers to your email server connection properties which are defined as a comma separated list of strings.
- The first parameter is the host of your email server which can be specified by hostname or IP address. It will look something like smtp.yourCompany.com for example or smtp.gmail.com
- The second parameter is optional and defines the port where the SMTP server is listening. Common ports are 25 and 587 but it could really be any other port depending on how your mail server is configured.
- The third and fourth parameters are also optional and are used to set a user and password to access your email server.
To properly get the email server connection properties, you will want to contact your IT department and describe what you are trying to achieve as many email servers are configured with strict security policies that will prevent the script from successfully connecting and using your corporate e-mail server. Folks in your IT department should know how to give you the right hostname and port for your email server. Handing over the source code of the Python script may be of help too.
If you want to configure this script using your Gmail, You will need to configure your Gmail account with 2-step verification, and then setup a Gmail App password. Once you have done that, use your complete Gmail e-mail address as the user (third parameter) and the App password for the fourth parameter. When using Gmail you do not need to specify a port number, so simply type "" in the second parameter.
- onemailflag: If you leave this parameter as 1, then an e-mail will be sent summarizing all records added since the Python script executed for the last time. If you switch the value to 0, then the script will trigger one e-mail for every record added.
- portalURL: You only need to change this setting if working against ArcGIS Enteprise. You will want here the full https location of your portal including the Web Adaptor.
- fsURL: This is the URL of your survey's feature service, which could be running in ArcGIS Online or ArcGIS Enterprise.
- fsLayerNum: This is the index of the layer in the feature service that you want to monitor. If you want to check for changes in the main layer of your survey, leave it to 0. If you want to check for changes in the layer of a repeat, change the index accordingly.
- serviceuser and servicepw: The ArcGIS credentials of a user with access to the feature service.
After saving changes to the init.json file, it is time to give the script a quick test:
- Run the script once so it can capture the current state of your feature service.
- Submit one record to your feature service and run the script again.
If any errors occur during execution, an error text file will be created in the same folder where you saved the Python script.
Scheduling the script
The Windows operating system includes a simple utility called Task Scheduler. It is quite easy to setup. Once you have determined when the script will be triggered for the first time, you can repeat its execution at regular intervals. For example, every 5 minutes. Obviously, the computer where you setup the task will need to be running all the time, although you can configure the task to run regardless of who is logged in.
The configuration of tasks in the scheduler is pretty much self-explanatory, but here are some specific instructions that can save you some back and forth:
- General: Check the option to run with highest privileges and set the task to run even if you are not logged-in.
- Trigger: If you want to quickly test your task, you can simply select your task in the gallery and then hit Run in the Selected Item panel on the right. When configuring the task for real, I suggest you select the startup trigger and that you also configure the task to run indefinitely every five or ten minutes or so.
- Actions: You will need to be particularly careful with this one. The Program/Script setting needs to point to the Python executable (Python.exe).
- If using your own copy of Python, refer to the installation directory of Python where you will find Python.exe
- If using Python from ArcGIS Pro, it will be under the Pro installation directory. For example: "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe"
You also need to indicate the location of your DetectChanges.py Python script as an argument. Do not forget to include the .py extension in the path. If the path to your DetectChanges.py file includes spaces, then you need to enclose the path with quotes. Lastly, set the 'Start in' property so it points to the directory in which you are storing the DetectChanges.py file.
Beyond the basics
The technique described in this blog post is a bit rudimentary, but it may do its job reasonably in some simple scenarios. You can manipulate the Python script to properly format the email message to be sent, and also to apply some logic to determine if a n email needs to be sent, and to who. Since you are in 'Python land' already, you can get creative and import the arcpy.mapping module to do all sort of sophisticated things in the script. You can for example take the incoming feature added into your feature service and do a straight (buffer) or network distance (closest facility) search to determine who will be notified.
Differences between the original script and this one
There are a handful of differences between the original Python script published in November 2017 and this one. For the most part, they are the same, but here are the key changes:
- The most important difference is how changes in the feature layer are found. In the November 201 script the script exercises a new API introduced in ArcGIS Online to detect changes. This new API only works in ArcGIS Online (not in Enteprise) and it had to be enabled manually on the survey's feature service. The latest version of the script, described in this post, uses Editor Tracking. Editor Tracking is enabled by default in every feature service created by Survey123 for ArcGIS. Editor Tracking is compatible with ArcGIS Online feature services and also with ArcGIS Server feature services back to 10.1.
- One email per feature or one per execution: The other key difference is that in our November 201 version, it was not possible to easily configure the script to aggregate all changes in a single email. With this version, you can use the 'onemailflag' parameter in the init file to decide if you want to send one email per added record, or one email per execution of the script.
If you are already using the November 201 version of the script and want to switch to this one, I recommend that you copy the Python script and the new init file into a separate folder. The init file is different, so pay attention to the new parameters introduced.