Latest Contributions by IsmaelChivite

This update to Survey123 is in tribute to our Esri Beijing R&D Center, which leads the vision and development of the Survey123 website and Web Forms. Survey123 for ArcGIS is developed by a small (but passionate ) team spreading over Redlands (USA), Melbourne (Australia) and Beijing (China). Coinciding with GIS Day, on November 15 2017, our Beijing friends put together a field exercise at the Mutianyu section of the Great Wall.  We headed up, capturing data with Survey123 all the way to Tower 26. It was an unforgettable experience! The  Great Wall of China spans over more than 10,000里 (5,000km). It is  certainly one of the most impressive architectural feats in history.        Better Web Forms The significance of this release is the introduction of a much improved support for Web Forms. Web Forms let you capture data with the surveys you design in Survey123, from web browsers. Web Forms are handy because they run well on desktop and mobile devices, not requiring the end user to download anything extra in order to capture data. Web Forms can also be easily shared through a simple link in an e-mail, they can be embedded within a web site or shared via social media. We see more and more people authoring smart forms with Survey123 for use within a web browser.  Typical use cases include campaigns to gather data from the general public, citizen science projects but also forms for routine work within the enterprise. You can open any survey you have published from a web browser by simply using the links provided in the Collaborate tab of the Survey123 web site. The above will not take you by surprise, because we have supported Web Forms since June 2016. So what is new? Web Forms authored from Survey123 web designer are faster: If you use the web designer to create your own surveys, you will find a noticeable boost in performance. This counts, because  particularly for public surveys people expect an instantaneous response when loading the survey.  We have reduced significantly the total size of the surveys. The  more lightweight your survey, the less time it takes to download into your web browser.  Most importantly, surveys have been improved so the browser can start rendering the questions before the full survey is downloaded.  This reduces the perceived startup time significantly. In our own testing, surveys with over 50 questions that used to take 11 seconds to load in the web browser, now only need 2.5 seconds. This performance boost will only take effect with surveys that you publish with version 2.6. That is, starting today...  If you already have published your survey and you want to take advantage of the performance improvements in this release, simply go into the web designer and re-publish your survey. If  you already have published surveys with web designer and want to take  advantage of the performance improvements in this release, simply go  into web designer and re-publish your survey.                                   General release for Web Forms authored from Survey123 Connect: If you follow Survey123 development closely, you will know that until this release we supported surveys from Connect on the web as a Beta feature. As of this release, it is not Beta anymore! A lot of work went into fixing critical issues and improving the user experience so you can author surveys in Connect, and then have people use them from a web browser. While there are still some known limitations, we feel confident that Web Forms now are solid to support the more common advanced features that Survey123 Connect offers.  This includes additional question types not currently available in web designer, as well as your own custom logic for user-input validation, conditional statements, repeats, groups etc Not everything that is possible in Survey123 Connect will work in a Web Form, but most will. The following table provides a quick reference to help you identify which XLSForm features are not supported in Survey123 Web Forms. XLSForm features not supported in Survey123 Web Forms Question types All XLSForm question types supported in Connect are supported except barcode. Appearances All appearances supported except spike and spike-full-measure supported (more on these appearances in a couple of weeks...) Other XLSForm Columns body::esri:inputMask and bind::saveIncomplete are not supported. XLSForm expressions pulldata() function is not supported. can't set the background image in questions of type image with the annotate appearance property() function is not supported (although you can use the email and username question types to achieve the same) Please also note that the look and feel of Web Forms may differ (although no drastically) from the way the survey is rendered in the Survey123 field app. Overall, the experience will be similar, but we have made conscious decisions to optimize data capture in the web versus the native app. Web Forms in this release also come with refinements in the user experience of various question types, particularly questions to capture locations, images/photos and single and multiple choice questions as well as many, many software defects fixed. The release of this new Web Form engine involved a pretty big undertaking so we are really excited about you finally having access to it. Enhanced Report Print Templates (Beta) As you may already know, Survey123 includes a handy built-in reporting engine that allows you to define the look and feel of reports using templates built with Microsoft Word.  Once the template is associated with your survey, you can easily create high quality printable documents using data submitted to Survey123. This work was first introduced through Tega's blog post Leveraging Custom Form Report Templates in Survey123.  The report service is at the moment exposed as a Beta feature and we continue evolving it with your feedback. In this release we have simplified and expanded the syntax used in Microsoft Word document templates as follows: Images: Up until 2.6, you could include images in your Microsoft Word template using the keyword ${%image1}, where image1 would be the name of the question in your survey with the image.  As of this release, you no longer need to prefix the question name with the percent sign %. That is, ${image1} will do!  This makes things easier for all of us. Most importantly, you can also specify the exact size of the image in your document. For example, you can set the width of your image in the final report to 300 pixels by using ${image1 | size:300:0}.  You can specify both the width and the height, but if  you leave the height set to 0, we will ensure that the aspect ratio is preserved. That is, we will guarantee that the image in your document has the width you specify, and then we will set  the height  accordingly to preserve the aspect ratio of the original photo. You can also now retrieve metadata from your images and include it as part of your report. Here is the syntax:   ${image1 | getValue:"x"} and ${image1 | getValue:"y"} will give you the X,Y coordinates from which the photo was taken. ${image1 | getValue:"date"} and ${image1 | getValue:"time"} will extract the date and time when the photo was taken. ${image1 | getValue:"name"} returns the name of the file. Many thanks to Peters, Amy‌ and all who voted for Include photo coordinates on reports from Survey123                      Maps: Simplified grammar so you can simply say ${Location} instead of ${%Location}.  Many of you have requested the means to define not only the scale and size of the map, but also the Web Map to be used.  This now can be achieved through the mapSettings optional parameter as follows:  ${Location | mapSettings:webmapID:mapScale}.  For example: ${Location | mapSettings:ffb2f300ceeb417e9b384f52b60ce4e4:10000 | size:600:400} will include in your report a map centered at the geometry found in the Location question (this is expected to be a geopoint question), using a Web Map with ID ffb2f300ceeb417e9b384f52b60ce4e4, at a scale of 10000 and with a size of 600 pixels in width and 400 pixels in height.  If you do not remember the name of the geopoint question in your survey, you can also use the keyword $shape.  For example: ${Location | mapSettings:ffb2f300ceeb417e9b384f52b60ce4e4:10000 | size:600:400} is equivalent to ${shape | mapSettings:ffb2f300ceeb417e9b384f52b60ce4e4:10000 | size:600:400} Multiple choices: With this new release you can decide how to include in your report the information contained in multiple-choice questions. For example, if you use ${multichoice1}, your Word document will include all selected choices in a single-line and separated by commas. If you choose to use ${multichoice1 | appearance:"bullets"}, then your Microsoft Word document will output the user selection as a bulleted list. All the above new grammar is in addition to everything supported so far and described in the Print Individual Responses help topic.  It is also in addition to the old-style grammar you used before, so if  you have already uploaded your own report templates, they will continue to work as usual. We will continue listening to your feedback regarding this beta feature and evolve it accordingly. Survey123 web site The Survey123 website includes some additional fixes and enhancements. I want to highlight a number of them: BUG-000109826 In Individual response window on Survey123 for ArcGIS website, leading zero is dropped for a selected record if the survey form contains values with leading zero in label column. BUG-000109752 When using Survey123 for ArcGIS web app, users are unable to scroll through the 'Thank You' screen if the message after submitting a survey does not fit within the web browser display BUG-000108470 Individual Response fields appear blank for a selected record in the Data tab of the Survey123 for ArcGIS (survey123.arcgis.com) website if the submission contains a leading zero BUG-000107912 Survey results exported to PDF from survey123.arcgis.com show the tags if html was used to style the survey with multiple selections ENH-000104866 Provide URL parameters to apply to fields for web surveys from Survey 123 online BUG-000110685: Date values do not get submitted from Web Forms published from Connect in Safari and IE11 browser BUG-000109871: HTML tags added to choices tab in XLSForm are displayed in Survey123 website Survey123 Field App and Survey123 Connect for ArcGIS The Survey123 field app and Connect have been updated to version 2.6. You can download the field app from the Apple, Google Play and Amazon app stores.  As usual, you can also download the desktop version of the app and Connect from our Downloads Page.  This is a minor release including a few fixes and minor enhancements: BUG-000106527 / DE-000002011: Date & Time cannot take selected values. A usability fix for the Date and Time controls. BUG-000103945 / DE-000001638: Android Photo Gallery. You can now browse for photos in Android devices using a nice photo gallery. BUG-000109192: Save survey in Draft option is not working when the survey form is created with repeat_count value. BUG-000107386: Hints cut off all text after a less than symbol '<' in Survey123 for ArcGIS field app. BUG-000105455: Home button on the Inbox (and Drafts) map overview ignores the custom home location. Eliminated conditions that caused the field app to occasionally crash after a new survey download. Clearing the Inbox and Sent folders in one survey will no longer clear Inbox and Sent folders in other surveys downloaded in the device. "time" type calculations not refreshing  Improvements to the camera flash on Android devices. Improved zoom capabilities in the barcode scanner. For more details on What is New in this release as well as all previous updates, refer to our What's New Help Topic. ... View more

Back in November 2016, I published a blog post describing how to automate e-mail notifications against a Survey123 project. The idea was simple: A Python script that would trigger an email  every time a new survey is submitted. By scheduling the script to run unattended at certain time intervals, one could easily put together a simple e-mail notification system. I want now to revisit this topic and share a new Python script, which takes care of some limitations of the first version. Specifically, the new script: Works against ArcGIS Enterprise in addition to ArcGIS Online Can be configured to either send one email for every newly added survey, or alternatively send one email for all new records added since the script was last launched. While the Python script can be used as is, I really share it as a starting point from which you can write the exact logic that matches your needs. You may for example tweak the script to: Format the e-mail notification following specific branding guidelines. Extend the notification logic so the recipient of the email varies depending on data submitted. Below are step by step instructions so you can configure this Python script with your own survey. Preparing your computer to run the Python script Technically, you can host the script in any computer that is able to run Python 3.x, however, you will want to select the host carefully. You will want  the script to run regularly and unattended, so you want to pick a computer that will reliably running 24x7 and constantly connected to your network. In addition to Python 3.x, the script requires the 'requests' Python module installed.  If your computer already has ArcGIS Pro installed, you already have Python 3.x and the 'requests' module installed. However, ArcGIS Pro is not required to run this script. How to install Python How to install the 'requests' module Preparing your feature service The Python script works against your survey's feature service. The script requires that your feature service has Editor Tracking enabled. By default, all feature services created by Survey123 have Editor Tracking enabled, so there is really nothing that you need to do extra. If  your survey is configured to work against a service from ArcGIS Server, then you will want to make sure  Editor Tracking is enabled, and if not, enable it manually. If you do not enable Editor Tracking, the  script will generate an error  text file indicating that Editor Tracking needs to be enabled and will stop execution. Downloading and configuring the Python script Now you can download and configure the script as follows: Download the Detect Changes and Notify Python script v2 and its associated init.json file and save them together within a folder in your computer. Edit and save the init.json file. At the very least, you will need to change the properties highlighted  in blue below.  {    "email":{      "recipients":["[email protected]"],       "from":"[email protected]",       "subject":"New features has been added to your service",       "text":"Hi Team,\n\nYou are receiving this e-mail because features have been added in:\n",   "server":["smtp.acme.com", "", "", ""], "onemailflag":1    },    "service":{       "portalURL":"https://www.arcgis.com", "fsURL":"https://services2.arcgis.com/fJJEXNgxjn0dpNsi/arcgis/rest/services/service_20dc5e3eab3f4efab8fde88f6027911b/FeatureServer",       "fsLayerNum":0,   "serviceuser":"yourusername", "servicepw":"yourpassword",       "fieldstoreport":["*"],       "viewerMapLevel":19    },    "filenames":{       "lasteditfile":"lastedit.json"    } } recipients: This is the e-mail that will be receiving notifications. You can create an email alias if you want several people to receive the email. from: This is the e-mail of the person/organization that is sending the e-mail. You can also set it to a DoNotReply email address. server: This setting refers to your email server connection properties which are defined as a comma separated list of strings.  The first parameter is the host of your email server which can be specified by hostname or IP address. It will look something like smtp.yourCompany.com for example or smtp.gmail.com The second parameter is optional and defines the port where the SMTP server is listening. Common ports are 25 and 587 but it could really be any other port depending on how your mail server is configured. The third and fourth parameters are also optional and are used to set a user and password to access your email server. To  properly get the email  server connection properties, you will want to contact your IT department and describe what you are trying to  achieve as many email servers are configured with strict security  policies that will prevent the script from successfully connecting and  using your corporate e-mail server.  Folks in your IT department should  know how to give you the right hostname and port for  your email server.  Handing over the source code of the Python script may be of help too. If you want to configure this script using your Gmail,  You will need to configure your Gmail account with 2-step verification, and then setup a Gmail App password.   Once you have done that, use your complete Gmail e-mail address as the  user (third parameter) and the App password for the fourth parameter.   When using Gmail you do not need to specify a port number, so simply type "" in the second parameter. onemailflag: If you leave this parameter as 1, then an e-mail will be sent summarizing all records added since the Python script executed for the last time. If you switch the value to 0, then the script will trigger one e-mail for every record added. portalURL: You only need to change this setting if working against ArcGIS Enteprise. You will want here the full https location of your portal including the Web Adaptor. fsURL: This is the URL of your survey's feature service, which could be running in ArcGIS Online or ArcGIS Enterprise. fsLayerNum: This is the index of the layer in the feature service that you want to monitor. If you want to check for changes in the main layer of your survey, leave it to 0. If you want to check for changes in the layer of a repeat, change the index accordingly. serviceuser and servicepw: The ArcGIS credentials of a user with access to the feature service. After saving changes to the init.json file, it is time to give the script  a quick test: Run the script once so it can capture the current state of your feature service. Submit one record to your feature service and run the script again. If any errors occur during execution, an error text file will be created in the same folder where you saved the Python script. Scheduling the script The Windows operating system includes a simple utility called Task Scheduler. It is quite easy to setup. Once you have determined  when the script will be triggered for the first  time, you can repeat its  execution at regular intervals. For example, every 5 minutes. Obviously, the computer where  you setup the task will need to be running all the time, although you can configure the task to run regardless of who is logged in. The configuration of tasks in the scheduler is pretty much self-explanatory, but here are some specific instructions that can save you some back and forth: General: Check the option to run with highest privileges and set the task to run even if you are not logged-in. Trigger: If you want to quickly test your task, you can simply select your task in the gallery and then hit Run in the Selected Item panel on the right.  When configuring the task for real, I suggest you select the startup trigger and that you also configure the task to run indefinitely every five or ten minutes or so. Actions: You will need to be particularly careful with this one. The Program/Script setting needs to point to the Python executable (Python.exe). If using your own copy of Python, refer to the installation directory of Python where you will find Python.exe If using Python from ArcGIS Pro, it will be under the Pro installation directory. For example: "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" You also need to indicate the location of your DetectChanges.py Python script as an argument. Do not forget to include the .py extension in the path. If the path to your DetectChanges.py file includes spaces, then you need to enclose the path with quotes. Lastly, set the 'Start in' property so it points to the  directory in which you are storing the DetectChanges.py file. Beyond the basics The technique described in this blog post is a bit rudimentary, but it  may do its job reasonably in some simple scenarios. You can manipulate the Python script to properly format the email message to be sent, and also to apply some logic to determine if a n email needs to be sent, and to who.  Since you are in 'Python land' already, you can get creative and import the arcpy.mapping module to do all sort of sophisticated things in the script. You can for example take the incoming feature added into your feature service and do a straight (buffer) or network distance (closest facility) search to determine who will be notified. Differences between the original script and this one There are a handful of differences between the original Python script published in November 2017 and this one.  For the most part, they are the same, but here are the key changes: The most important difference is how changes in the feature layer are found. In the November 201 script the script exercises a new API introduced in ArcGIS Online to detect changes. This new API only works in ArcGIS Online (not in Enteprise) and it had to be enabled manually on the survey's feature service.  The latest version of the script, described in this  post, uses Editor Tracking. Editor Tracking is enabled by default in every feature service created by Survey123 for ArcGIS. Editor Tracking is compatible with ArcGIS Online feature services and also with ArcGIS Server feature services back to 10.1. One email per feature or one per execution: The other  key difference is that in our November 201 version, it was not possible to easily configure the script to aggregate all changes in a single email. With this version, you can use the 'onemailflag' parameter in the init file to decide if you want to send one email per added record, or one email per execution of the script. If you are already using the November 201 version of the script and want to switch to this one, I recommend that you copy the Python script and the new init file  into a separate folder. The init file is different, so pay attention to the new parameters introduced. ... View more