Skip navigation
All Places > Survey123 for ArcGIS > Blog > Author: ichivite-esristaff
1 2 3 Previous Next

Survey123 for ArcGIS

91 Posts authored by: ichivite-esristaff Employee

If you use Survey123 Connect to author forms you are surely to become a regular of Microsoft Excel and the Survey123 XLSForm templates and samples. In this blog post, you will find some tips to help you be more productive with all of that. The first half of the post focuses on the Survey123 XLSForm templates and samples, describing how you can manipulate them to your liking. The second half compiles a handful of general productivity tips for Microsoft Excel.

 

Survey123 Connect XLSForm Templates

 

Some context first: XLSForm, .xls files and Microsoft Excel

 

Before jumping into the actual tips, I want to provide some clarification about Microsoft Excel and how it relates to XLSForm and Survey123. The XLSForm specification describes the syntax you need to follow in order to define a smart form in an .xls spreadsheet file. Survey123 adheres to the XLSForm specification. The foundation of Survey123 forms is ultimately an XLSForm document (.xls file).

 

Xls files were introduced in 2007 by Microsoft. Initially it was a proprietary format, but it was ultimately shared as an Open Specification. Technically, you can use any means to create Survey123 forms as long as you follow the XLSForm syntax and the output is stored in a proper .xls binary file. You do not need to use Microsoft Excel, other programs such as Open Office, Libre Office, Numbers can create xls files.

 

Formulas and data validation in Survey123 templates and samples

 

Survey123 Connect includes a number of samples and templates to help you get started with XLSForms. These .xls files (technically .xlsx) have been created using Microsoft Excel, and while we know they can be used and manipulated outside Microsoft Excel, we do not test them outside of this program.  On top of the compulsory XLSForm sheets (survey, choices and settings), our templates and samples include a number of formulas and data validation rules meant to facilitate the authoring of XLSForm compliant documents.  For example:

 

  • Several XLSForm columns in the survey spreadsheet include choice lists: type, appearance, bind: esriFieldType, etc.  This is to make it easier for you to pick a valid XLSForm choice.
  • A data validation rule in the name column of the survey sheet will highlight cells that have duplicated values. It also ensures that the value you enter will be accepted by ArcGIS as a field name. It checks for reserved keywords, length and special characters.
  • A yellow tip briefly describes the intent of each XLSForm column.
  • etc.

 

The formulas and data validation rules included in the Survey123 templates and samples are for your convenience, but they are not strictly necessary. If you feel these validation rules and formulas are in your way, you can manipulate and even eliminate them.

 

The next animation shows how you can access the data validation rules of a cell in Microsoft Excel.

 

 

Virtually every cell in our samples and templates has data validation. At this moment, I just want to make sure you know how to open the data validation rules for a particular cell.

 

Yellow tips (and how to hide/disable them)

 

When you select a cell in the survey sheet, you always get a yellow tip.  We added them to describe the purpose of the different columns in the survey sheet.  Below is the tip for the bind::esri:fieldAlias for example.

 

 

The yellow tips provide good info, but they also can get on your way obstructing the view of your form design. If you simply hit the Esc key, the tip goes away until you select another cell.

 

 

The yellow tip, in Microsoft Excel jargon, is an Input Message. To disable it, you need to go into the data validation dialog as shown in the next animation. 

 

 

If you feel ready for it, you can also disable all the Input Messages in one go. First, select all cells in the survey sheet using Ctrl-A, then open the Data Validation dialog and disable the Input Message for all.

 

Manipulating XLSForm lists such as appearance, type, etc.

 

As described above, some specific columns in the survey sheet limit inputs to values within a list. This is the case, for example, for the type, appearance, bind::esri:fieldType, required and readonly columns. It is important to know how to manipulate these lists.

 

Take the type and appearance lists, for example. They define, respectively, the XLSForm question types and question appearances that Survey123 supports. As support for new question types and appearances grows, lists in old surveys  you may have created could become obsolete.  For example, support for geotrace and geoshape question types came with version 3.6.  If your survey used a template downloaded with 3.5, these question types will be missing.

 

If you look at the data validation of a cell in the type column, you will see that it limits input to a list. The list is defined as a range of cells coming from the types sheet.  The types sheet is not part of the XLSForm spec.  It was spcifically added to the Survey123 Connect templates to help build data validation rules like this. It also serves as a nice quick reference for XLSForm syntax, listing the different question types, appearances, operators  and other things you can use.

 

If you want to add a missing entry to a list, you will do it in the types sheet. The next animation shows how you can add the geotrace and geoshape question types into a pre-3.6 template.

 

 

The same technique described above can be used to add new appearances. 

 

You should keep in mind two important things:

You must add new entries in the types sheet one by one. That is: You insert one new row, and you add values in it. Then you add the next row and so on and so forth. If you insert more than one row without values, your list range will not be happy.

You must enter new entries somewhere in the middle of the list. That is, never add a new entry at the very beginning (first position) or end (last position) of the list.

In the case of the lists for the required and readonly columns, you may want to delete the data validation rule completely if you want to enter an XLSForm expression in them.  The list included by default only has one value: yes. If you want to enter expressions you may want to delete the data validation rule to eliminate the drop-down list.

 

Creating your own template

 

By now you may be wondering if you can create your own template. To some extent, the answer is: yes. You can first create a new survey from the Advanced Template and make your own modifications to the Excel file. You can then use the Create a new survey option to derive new surveys out of your own template.

 

 

What I like about this approach is that not only you can create a new survey with all the modifications in your Excel file, your new survey will also inherit other properties of your template, such as the thumbnail, summary and description of your own template. Style, Inbox and Image settings in your template will also be matched, as well as custom content in your media folder if you ever add it to your template form.

 

If you publish your template from Survey123 Connect and share it with groups within your organization, other people will be able to use it  too.  Your survey, once published, will appear in the My Organization category of the New Survey dialog in Connect.

 

The real advantage here, again, goes well beyond tweaks to the data validation rules and formulas. It is other things in your template, like settings or even common lists you may have added in the choice sheet that would benefit anyone in your organization using your template.

 

Microsoft Excel Productivity Tips

 

So far, I have described specific tips to handle the Survey123  templates and samples that come with Survey123 Connect. Next, I want to compile a few general Microsoft Office tips for all of us who spend time designing Survey123 forms using XLSForms, Microsoft Excel and Connect. It is likely that you will already know many of the tips listed, but if just a single one you don't, that may make this read worthwhile.  If you have other tips you want to share, please add them as a comment below, so we can keep the list growing.

 

Inserting new rows, the quick way: Ctrl-Shift-+.

 

It is no secret that you can insert a new row by selecting a row, right-clicking and then using the insert option in the context menu. If you want to do that same thing much more quickly, select a row and then hit Ctrl Shift +.

 

 

Careful with copy paste!

 

Copy-pasting is often our friend. Nothing wrong with it... unless you copy too much. In Excel, when you do a standard copy-paste, you actually are copying way more than you think: you are copying the cell values, but also formatting style and formulas.  It is the formula bit that can get you in trouble, particularly when you copy cells from one xls file to the next.  To avoid any issues, I like to copy just the values.  This is described well in this Microsoft article.

 

Extend values and formulas

 

This tip is particularly valuable when constructing lists in the XLSForm choice sheet. Simply double-click on a cell's bottom right corner to extend that value and formula down to the end of the list. In the animation below this technique is used to apply a list name to over 250 list choices.

 

 

Keyboard shortcuts

 

In Microsoft Excel, using the mouse is often the slowest approach to get anything done. There are keyboard short-cuts for anything you can imagine. I use keyboard shortcuts for selections quite a bit: Ctrl + A to select all cells in a sheet and  Ctrl + Shift + Down Arrow to expand a selection to the bottom of a list.

 

But there are way too many shortcuts to describe and only a few that you will truly take advantage of. If you hit Alt, Microsoft Excel will describe keyboard shortcuts to open a ribbon. For example, Alt + A opens the Data ribbon but you do not have to remember that, because if you simply hit Alt, the Data ribbon will be highlighted with A as a reminder. Even better, if you hit Alt + A not only the Data ribbon opens, but also every function within it is also highlighted with its shortcut. For example, Alt + A + V opens the Data Validation menu.

 

 

Again, just a few tips. If you want to share a few more, please do so below!

There is truth in the saying a picture is worth a thousand words, but sometimes I wonder if that is enough, no pun intended, to give you the whole picture. Lets burn into that image the date and time when the photo was taken, the geographic coordinates of that location or some other critical information, and the value of that picture is now multiplied. That will make your photo be worth a few more than one thousand words, and most importantly, give you the whole picture.

 

Watermarking is an old, yet useful concept, allowing you to superimpose extra information on a photo.

 

In the Street Service Report survey example in the right side, you will appreciate that the photo taken from Survey123 automatically includes the Job ID as well as the date and time when the photo was taken.  

This information is part of the image, it is a photo watermark.

 

Watermarking is often used for two main reasons:

 

  • To add critical information necessary to understand what is depicted in the photo, even if the photo is presented out of context.
  • To show evidence that the photo was taken at a particular time and location, or by a particular person.

 

Many field data collection workflows can benefit from photo watermarking. Asset inspections, damage assessments or code violation reports are good examples.  For an asset inspection, you may want to burn into the photo the unique identifier and status of the asset. In a damage assessment or while documenting a code violation, you can use watermarks to carry the location, time and name of the inspector or team that shot the photo.

 

Starting with version 3.5, the Survey123 field app has built-in photo watermarking capabilities.  As you will learn through this article, you can create watermark expressions to add date, time, location and user information into your photos. You can even dynamically set the content of your watermarks with data previously entered in your smart form.  Survey123 watermark expressions give you a great deal of flexibility to control the placement and display properties of your watermarks.  All it takes is a bit of XLSForms magic.

 

To configure a survey with photo watermarking capabilities, you need to design your survey with Survey123 Connect. Survey123 Connect is a desktop application used to author Survey123 projects following the XLSForm specification. With XLSForms you define the questions and behavior of your smart form in Microsoft Excel (following the XLSForms syntax). XLSForm files are then published as Survey123 forms into ArcGIS with Survey123 Connect. If you are not familiar with Survey123 Connect and XLSForms, I suggest you start with our XLSForms video tutorials in YouTube, or with our step by step XLSForm guides.

 

Getting started with photo watermarking

 

Watermark definitions are applied to image questions using the bind::esri:parameters XLSForm column. For example:

 

typenamelabelbind::esri:parameters
geopointlocationGPS Location
imagephotoPhotobottomRightWatermark="Bula World"

 

The above will add a watermark with the text Bula World in the lower-right corner of your photo. Pretty much self explanatory. Don't you think? The most obscure aspect in all of this is the Bula part.

 

A photo watermark expression in Survey123 is split by an equal sign into two parts:

 

  • Watermark placement: The first half defines the placing of your watermark within the photo.
  • Watermark content: The second part is always enclosed in double quotes and defines the contents and display properties of your watermark.

 

Watermark placing

 

The first half of your watermark expression defines its location within the image. You can choose betweeen 9 different positions:

 

  • topLeftWatermark
  • topCenterWatermark
  • topRightWatermark
  • leftCenterWatermark
  • centerWatermark
  • rightCenterWatermark
  • bottomLeftWatermark
  • bottomCenterWatermark
  • bottomRightWatermark

 

For the bottom-right corner of your photo, you can either use bottomRightWatermark, or simply watermark, for short. That is, bottomRightWatermark="Bula World" and watermark="Bula World" are equivalent.

 

You can add more than one watermark to your photos. For example, you can add one in the top-left corner, and another one in the bottom-right corner.  We will look at this later but before that, lets progressively disclose how to control the contents of your watermark.

 

Adding static text and basic text formatting

 

The content and display properties of your watermark are set in the second half of your watermark expression.

bottomLeftWatermark="Bula World&color=red"

leftCenterWatermark="Bula World&color=red&size=18"

watermak="Bula World&color=red&outlineColor=black&haloColor=white&size=12&bold=true&margin=10"

 

Note that you can add one or more formatting parameters. All you need to do is to separate them with an ampersand sign (&). The complete list of text formatting parameters can be found in the Watermark Hep Topic.  Here are the most common:

 

ParameterExampleNotes
color

color=red

color=#0000FF"

You can specify the color using a color code name or an hexadecimal value. The default color is blue. Find a list of colors at W3Schools.
outlineColoroutlineColor=blackHexadecimal color values and color names are also accepted.
haloColorhaloColor=white
fontfont=ArialArial, Helvetica, Cochin, Scheme, Verdana, Optima, Courier New...
sizesize=12Set in pixels
marginmargin=10Sets the margin in pixels from the edge of the image

 

Colors: In terms of colors, I like to set a bright color for the text (color), a dark one for the outline and then white for the halo. That ensures the watermark will stand out on top of your photo. The W3Schools list of color codes and hexadecimal values is a great place to help you define colors properly.

 

Text (and image) Size: The size parameter deserves a more detailed explanation.  Setting the right size is going to be a trial and error exercise and will largely depend on how you plan to use your watermarks.  For example, you may want to watermark photos so you can use them in a printed report, or simply show them in a screen within a dashboard or web mapping application. Depending on the use, you may want larger or smaller text.  Testing the size of your text in Survey123 Connect can be tempting, but you really want to look at your watermarks in the same environment where your users will ultimately use them. Do not go by what you see in Connect: validate your text size on paper if you intend to print, on a mobile device if you want people to preview the watermark from a device, etc

 

The size parameter is measured in pixels. For this reason, the size of the photo really comes into play.  While there are many devices out there and each may generate photos of different sizes, Survey123 allows you to define the size of photos taken with the Survey123 camera.  This is controlled through the Settings | Images dialog in Connect as shown below.

 

 

By default, Survey123 Connect always sets your image size to 640px on the longest edge.  In a 640px image, text at 12 pixels in size will show pixelated.  You could increase the size to 16 pixels for better results. If you change the image size to 1280, at 12px your text will be very small; you may want to change then the size to 24 or 26.

 

The default image size set by Connect (640px) is not ideal for watermarking. Give it a go in your own, but I typically use 1280px because otherwise there is barely any room for the watermark.

 

Breaklines: If you want to break your text into two or more lines, you can simply add \n to your text. For example:

watermak="Bula \n World"

You will likely use break lines all the time to split your watermark content.

 

Adding dynamic content: Location and Dates & Times

 

Very often you will want to add information in your watermark about the current date, time and location where the photo was taken. You may be tempted to use XLSForm variables to get this information into your watermark, but you should not. We will learn later how to add data from your form into the watermark.

 

To add the current location or time information you will want to use watermark-specific syntax. For example:

watermak="Inspected on: @[dateTime:short]"

watermak="Inspected on: @[dateTime:short] \n At @[latitude longitude]"

watermak="Inspected on: @[date:dddd] \n At @[latitude longitude] \n Accuracy: @[accuracy] meters"

watermak="Lat @[latitude] \n Lon: @[longitude]"
watermak="USNG: @[usng]"

 

Here is a list of accepted placeholders to extract location-related information. All location data is relative to the place where the photo is taken.

 

PlaceholderNotes

@[longitude:ddm]

@[longitude:dms]

@[latitude]

@[longitude]

@[latitude longitude]

Use the ddm and dms qualifiers to get the coordinates in degrees-decimal-minutes versus degrees, minutes and seconds. If you do not explicitly set a coordinate format you get DMS.

You can use @[latitude longitude] to get the coordinate pair at once, but it is useful to get the coordinates separate so you can split the data across two lines.

@[mgrs]

@[usng]

@[utm]

Reports the location where the photo has been taken in MGRS, USNG and UTM coordinate formats respectively.
@[altitude]Returns the elevation in meters where the photo was taken as reported by the device.
@[compass]Returns the compass bearing at the time the photo was taken.

@[direction]

@[speed]

Returns the direction and speed of travel when the photo was taken. If the user is not moving when capturing the photo, these values will be empty
@[accuracy]The horizontal accuracy as reported by the device when the photo was taken.

 

Lets go with time-related information, which is also relative to the instant when the photo is taken.

 

PlaceholderNotes

@[datetime]

@[datetime:short]

@[date]

@[time]

Returns the date and time when the photo was taken. Unless otherwise specified, the date and time are formatted according to the date and time formatting rules in the device.  The :short qualifier will output the date/time in a compact format, but still using the formatting rules from your device.

 

If you want to dictate date/time formatting rules independent of the device used to capture the photo, then you need  to explicitly set formatting rules in the watermark expression.

 

Here are a few examples for formatting dates and time:

 

@[datetime:d MMM yyyy HH:mm]  Example output: 7 Aug 2019 14:01

@[date:dddd]                                  Example output: Sunday

@[time:hh:mm t]                              Example output: 14:23 Pacific Daylight Time

 

To create your own formatting rules, the following tables should come handy:

 

FormatterDescription
dThe day as number without a leading zero (1 to 31)
ddThe day as number with a leading zero (01 to 31)
dddThe abbreviated localized day name (e.g. 'Mon' to 'Sun')
ddddThe long localized day name (e.g. 'Monday' to 'Sunday')
MThe month as number without a leading zero (1 to 12)
MMThe month as number with a leading zero (01 to 12)
MMMThe abbreviated localized month name (e.g. 'Jan' to 'Dec')
MMMMThe long localized month name (e.g. 'January' to 'December')
YYThe year as two digit number (00 to 99)
YYYYThe year as four digit number

 

FormatterDescription
hThe hour without a leading zero (0 to 23 or 1 to 12 if AM/PM display)
hhThe hour with a leading zero (00 to 23 or 01 to 12 if AM/PM display)
HThe hour without a leading zero (0 to 23, even with AM/PM display)
HHThe hour with a leading zero (00 to 23, even with AM/PM display)
mThe minute without a leading zero (0 to 59)
mmThe minute with a leading zero (00 to 59)
sThe second without a leading zero (0 to 59)
ssThe second with a leading zero (00 to 59)
zThe milliseconds without leading zeroes (0 to 999)
zzzThe milliseconds with leading zeroes (000 to 999)
APThe AM/PM display. AP will be replaced by either "AM" or "PM"
apThe am/pm display. ap will be replaced by either "am" or "pm".
tThe timezone

 

 

Adding previously entered data from your form

 

It is possible to add a watermark using data previously entered in your form. In the example at the very beginning of this article, we added the Job ID value to the watermark.  You could also add an address or a priority, even some comments from the person completing the form.

 

Data from your form cannot be added directly in the bind::esri:parameters column. Do not attempt to include standard XLSForm functions within your watermark expression. The right way to do this is to first put together the watermark content in a separate question, and then reference that. Here is a basic example:

 

typenamelabelbind::esri:parameterscalculation
geopointlocationLocation
textcommentsComments
calculatewatermarkWatermarkconcat("Comments", ${comments},"&size=20")
imagephotoPhotowatermark=${watermark}

 

Note how the first half of the watermark expression defining the placement of the watermark is added in the bind::esri:parameters while the second half, defining the content, is prepared in a separate calculate question and then referenced through an XLSForm variable.

 

To compose the watermark content, I am using a calculate type of question. Calculate questions are never shown in the actual form, but are useful when you want to compute some temporary values to be used somewhere else within the form. The value is computed using an expression in the calculation column.  A calculate question is very similar to a hidden question.  The only difference is that calculations are compulsory for calculate questions and not for hidden questions.

If you do not want the output of your calculate to be stored as a GIS attribute in your feature layer, set its bind::esri:fieldType column to null.

The safest way to construct complex watermark expressions is through the concat() function.  You can do pretty interesting things when working in this way. Not only you can bring data from your form, but also control the formatting of your text dynamically. In the next example, we set the color and content of the watermark text dynamically based on information entered by the user in the form.

 

typenamelabelbind::esri:parameterscalculation
geopointlocationLocation
select_one yes_nois_urgentIs urgent?
textcommentsComments
calculatetext_colorText Colorif(selected(${is_urgent},yes),'red','green')
calculatewmWatermarkconcat("Comments", ${comments},"&color=",${text_color})
imagephotoPhotowatermark=${wm}

 

Finally, note that when you reference watermark content in bind::esri:parameters as shown above, you no longer need to enclose the second half of the watermark expression with double quotes. Double quotes are needed to help the parser process the contents of bind::esri:parameters in case that you include spaces in your watermark expression. When you reference pre-computed watermark content like in this example, you can't add spaces, so no quotes are needed.

 

 

Adding images to your watermark

 

Say you want to add a logo to your photos. You can. Simply add your image in PNG or JPEG format into the media folder of your survey, and then use the image and imageSize watermark parameters:

watermark="image=MyLogo.png"

watermark="image=MyLogo.png&imageSize=35&text=All rights reserved"

In the example above I wanted to show the text after the image, so I decided to add the text parameter explicitly.

 

Adding multiple watermarks

 

Now that you know how to create your own watermark expressions, you may like to hear that you can add many to the same photo!  All you need to do is to separate your different watermark expressions with spaces within the bind::esri: parameters column.

 

For example, you can write this within bind::esri:parameters:

topLeftWatermark="Bula World" topRrightWatermark="Bye World"

 

As with many things XLSForms, you will want to start easy and increase the complexity of your watermark expressions progressively. You can use Survey123 Connect to make sure your syntax is good and do some preliminary testing, but eventually you will want to test your watermarks from the field app and if you plan your photos to be printed, print them to make sure you get them right.

 

Known limitations

 

  • Photo watermarking is only available in the Survey123 field app. All watermark expressions will be ignored by the Survey123 web app.
  • You cannot combine watermark expressions with the annotate or draw appearances of image questions.
  • On iOS, the Unrestricted image size currently doesn't support watermarks. The image will be scaled down to 1920 pixels on the longest edge.

 

Other resources

 

A great resource to see many photo watermark examples is the Watermarks XLSForm sample included in Survey123 Connect.  It includes over 20 different watermarks across multiple pages.

 

 

Our Watermarks—Survey123 for ArcGIS | ArcGIS help topic is a great reference too.

 

 

Happy watermarking!

A new update to the #Survey123 field app is now available in the app stores. The new build number is 3.5.177. This minor update is a targeted release addressing two critical issues recently raised:

 

  1. BUG-000124227: Downloading a survey will hang when using Portal 10.7 or earlier, or when trying to download and open a shared survey via the share URL link from website (shortened URL).
  2. Offline TPK files are not displaying correctly in the basemap menu on 32-bit Android devices.

 

Many, many thanks to TCPUD GIS , Eric Pescatore , Lindsay King , Kenneth Eggering , Braden Burkholder , Carol Northcott , Tatag Bagus Narendra , Jennifer Lidgett , Mason Bailie and Luke Amos-Binks for helping diagnose the issues  above and validating the fixes through our Early Adopter Program.

A Survey123 update is now available across all supported platforms. This release includes some important fixes. To download the latest version of the Survey123 field app and Connect refer to the Install Survey123 for ArcGIS—Survey123 for ArcGIS | ArcGIS help topic. Changes to the Survey123 website are available without further action on your side.

 

Here are the main issues addressed:

 

Survey123 website

 

  • BUG-000123405 Cannot open a survey published with Survey123 Connect from the form item details page.
  • BUG-000123464 Generating Feature Reports in the Survey123 for ArcGIS website fails with the error message, "Generate reports failed. You do not have enough credits to perform this operation. Contact your organization administrator for assistance" when an organization has negative credits. Thanks to Becky N who described this problem in GeoNet:Feature Reports Failed - Not Enough Credits 

 

Survey123 Connect and Field App

 

  • BUG-000123925 A survey containing hidden question type storing the data values for start and end time of the survey (using calculation) fails to load the records in the Inbox folder.
  • BUG-000123746 Constraint validation fails to execute if used against a question set as non-required if the question is followed immediately by a hidden question with a calculation. Thanks to Tarun Jaiswal for raising this.
  • BUG-000123822 Pulldata (CSV) will not work against a question of type note if the XLSForm name column is left empty.  Thanks to Florence Raimbault for finding it out.
  • Date questions are now set to use a default time of noon as opposed to mid-night. This helps avoiding unintended time-zone shifting issues when applying XLSForm expressions. Kudos go to Nicolas Roger.
  • Geopoint calculations within a repeat are now working.Thanks to Adilson Furlani for reporting issues with calculations affecting the first record of a repeat with 3.5.
  • The page counter when using the pages style in a survey now works correctly in Arabic.

 

With this update, we are also introducing support for Garmin Glo 2. To learn how to make a connection to your Garmin Glo 2 from Survey123, please check Direct GNSS external receiver support in Survey123 3.3 

 

As usual, our What's new in Survey123—Survey123 for ArcGIS | ArcGIS help topic highlights what is new in this and previous releases.

 

In addition to this update, you will also find the initial Beta builds of our 3.6 release available for testing through our Early Adopter Community | Esri Early Adopter Program  The Beta builds include support for polygons and polylines, web maps, mobile map packages and many other map enhancements. A Windows setup program to install the Survey123 website and REST API locally is also ready for testing. We are looking at September for our next update.

Survey123 includes out of the box functionality to help you generate high quality documents out of submitted surveys. This is extremely useful when you need to present your survey123 data following strict formatting guidelines, for example when mimicking legacy paper forms or to comply with legal documents. 

 

In this blog post I will describe how you can configure this feature report functionality against your own surveys, defining the content and look & feel of your own feature reports.

 

 

A bit of context: Survey123 feature reports vs summary reports

 

Before we get into the details, I want to provide some context up-front about what exactly I mean by feature reports.

Generally, the word report is associated with summary reports where data is aggregated to help identify trends. Survey123 includes basic summary report capabilities, but these are not the focus here. This discussion is about feature reports. A feature report presents data from a single survey record.

 

Say for example you use Survey123 to support asset inspections and you want to generate one document, as proof-of-work, for each inspection performed. For that you use feature reports. Say Survey123 is used to document city code violations and you want to generate and mail individual tickets for violations observed... For that you use feature reports too.  The purpose of a feature report is not to aggregate data across multiple surveys. The purpose is to represent the data a single submitted survey.

 

In a way, the Survey123 feature report capability is about bringing your survey data back into a form format, except that in this case, you want the form to present data, as opposed to using the form to collect data.

 

Creating a feature report from the Survey123 website

 

The following animation shows how you would generate a report from the Survey123 website. Note that the feature report is triggered from the Data tab in the Survey123 website.  The Data tab is designed to help you explore and drill down into individual survey records (features).  Using the map or the table views you can select a particular survey record and generate a report for it. That is, of course, assuming that you have previously configured your survey with a template. You will learn shortly about creating your own report templates.

 

 

Feature reports can be generated for one or multiple records at once. You can use filters or manually select multiple records using the shift key.  If you create a single report, you will be able to open the document right away from your browser. If you create reports in bulk (from a selection), then the outputs will be stored in your ArcGIS account.

 

The Survey123 feature report service will always create a single document for every record you select. That is, reports from multiple records will not be merged into a single document.  If you select 200 survey records and you trigger feature reports for them, then Survey123 will create one document for each record: 200 separate documents.

 

Creating a Sample Report Template

 

To get started on your own, you need first a report template to create feature reports. This is how you create one:

 

  1. In the Survey123 website, go to the Data tab of your survey.
  2. Select one record from the table or the map.
  3. Click on Feature Report in the top bar
  4. In the Feature Report panel that just open on the left side, click on the Manage Templates link.
  5. Create a new sample template as shown in the animation below and close the dialog.
  6. Generate a report.

 

 

Now that you have a template, click on the Generate Report button and open the resulting output document.  The sample template is generated automatically, and as such, will not create the most beautiful results, but it is a great starting point for you to refine it.

 

Feature report templates are stored along with your survey in your ArcGIS account. If you navigate to the directory of your survey within your ArcGIS content, you will find your feature report Microsoft Word template in there.  You can associate one or more templates with a single survey.

 

All templates you associate with your survey will be available to any user that has Viewer access to your survey. That is, anyone who has been granted Viewer access to your survey, will also be able to generate reports using the templates you define for that survey.

 

Getting started with custom feature report templates

 

Going back to the Manage Templates link you will be able to download the sample template to your computer.  Once you download the template, you will realize that it is a simple Microsoft Word document. This is what the sample template for my Water Violation v2 survey looks like:

 

 

You can see right away how templates work: A series of placeholders for questions in your form are included in the template. The most basic syntax for the placeholders looks like ${question_name}. All you need to do is to make your Microsoft Word template document look the way you want and insert the Survey123 feature report placeholders accordingly. 

 

Once you are happy with your report template, you can upload it using the Manage Templates link.  Until you get familiar with the placeholder syntax, I suggest you start easy: Start by rearranging text within the document and copying and pasting the placeholders from the sample template. Change the color and style of the text, arrange existing placeholders within tables, etc.

 

Generate a report again with your updated template to see the effect.

 

Using advanced feature report syntax

 

Survey123 feature report syntax can get a bit more sophisticated than just a generic ${question_name}. For example, note in the example below how extra parameters are used to format dates, to control the size of a photo or signature and even to define the scale and contents of a map.

 

 

This is what the final output of the template above looks like when the report has been generated.

 

When working on your template, you first need to understand how to uniquely reference questions within your survey. Downloading the sample template is the easiest way to get a complete list of placeholders, but you can also get the complete list from the feature report template dialog as shown below.  This dialog also lets you explore some more advanced syntax for your placeholders.

 

 

You can insert a | sign after your question name (aka: field code) and include some extra parameters to either format the value written to the report, or extract other information from your survey question.

 

Feature Report SyntaxDescription
${myQuestion}Inserts  in your report the value stored in the specified survey question.
${myDateQuestion | format:"DD/MM HH:mm"}Date questions are tricky, because they will not come pretty unless you format them properly. The format parameter is your friend. For a complete reference look for the Formatting Dates section in our Formulas—Survey123 for ArcGIS | ArcGIS help topic.
${myPhoto | size:300:0}When working with image questions, you can set the exact size in pixels that the image will have in the output report document. The first value defines the width and the second the height. If you set the height to 0, then the width will be applied and the height will be dynamically set so your image aspect ratio is preserved.

${myPhoto | getValue:"name"}

${myPhoto | getValue:"date"}

${myPhoto | getValue:"time"}

Also for image questions, you can add to your report the photo filename using the getValue:"name" parameter. Similarly, you can also extract the date and time when the photo was taken and have it written in your report. That, of course, assuming that the EXIF of the photo includes that info.
${myLocat | size:400:400}The size parameter is valid for map questions too. In this case, we will add in the report  map with a size of 400x400 pixels.

${myLocat | mapSettings:"6712da572":5000}

${myLocat | mapSettings:"":5000}

Use the mapSettings parameter to describe the exact webmap you want to draw and the map scale.  First you pass the webmap item id and then, separated by a colon, the map scale. If you do not know how to get the webmap item id, have a look at Lisa Berry's  Where can I find the item ID for an ArcGIS Online item? blog post. If you pass an empty webmap item id, then the report will use a default webmap.

${myChoiceQuestion | selected:"yes"} Yes

When working with choice questions you can use the selected parameter to output a checkbox. The expression on the left will output Yes if the option selected is yes.  This works well with both checklists (select_multiple) and single choice and dropdown questions (select_one).

 

Combining your mastery of the syntax above and a bit of good taste applying layouts and formatting in Microsoft Word will go a long way. You can pretty much do anything you want with your Microsoft Word document. For example, you can change the size and orientation of the page, use tables to arrange content, set the alignment and justification of text, its color, size, etc.  You can also include Survey123 placeholders in headers, footers or anywhere in the body of your document.

 

As you get started with the feature report syntax, I suggest that you start slowly, making small incremental changes against the sample template and you let your confidence grow progressively. You will need to upload your template again and again until things look they way you want. Every time you upload the template, the syntax of your report template will be validated.  You will get errors when you add an incorrect number of parameters on your placeholders, when you reference field codes that do not exist in your feature service or when you do not open or close the brackets correctly.  Also be careful making sure parameters are set with the right casing and honor the necessary spaces.

 

On top of including photos, signatures and maps in your Survey123 feature report templates, you can also handle related records (repeats) and even define conditional visibility statements. The complete reference for the feature report syntax is included in our Feature report templates—Survey123 for ArcGIS | ArcGIS help topic.

 

ArcGIS Online support and credit consumption

 

Survey123 feature reports are generated through an ArcGIS Online premium service. At this moment, the use of this service does not consume ArcGIS Online credits, but it will starting with our next update to ArcGIS Online.

The feature report service will consume ArcGIS Online credits at a rate of 2.5 credits per document generated starting with the ArcGIS Online release in October of 2019.

            

In the event that a user sees the Feature Report functionality disabled in the Survey123 website, an administrator of the ArcGIS Online organization can enable the Feature report functionality for that user as shown in the animation below.

 

 

ArcGIS Enterprise support

 

The Survey123 feature report service is at this moment hosted in ArcGIS Online and it is designed to work with surveys hosted in ArcGIS Online. However, with some limitations, this service can also be used with surveys hosted in your own ArcGIS Enterprise 10.5 or newer.

 

  • Your ArcGIS Enterprise instance must be accessible to the Survey123 ArcGIS Online Feature report service. That is, you must configure your  ArcGIS Enterprise web adaptor to expose access to your Enterprise instance from the internet. Please note that portals with Integrated Windows Authentication will not work with the report service.
  • The inclusion of any type of feature service attachments (photo, signature, annotated images...) is not supported when your data is hosted in ArcGIS Enterprise. This is a limitation listed in the backlog of ArcGIS Enterprise.

 

Bearing the limitations above, the use of the ArcGIS Online Feature Report service from your ArcGIS Enterprise instance will not incur into any ArcGIS Online credit costs.

 

Automation

 

You can automate the generation of Survey123 feature reports through Integromat and the ArcGIS Python API.

 

  • Integromat: You can the Survey123 module in Integromat to automate the generation of feature reports when a new survey is submitted. Even better, with Integromat, you can also take that feature report document and include it as an e-mail attachment, or upload it into cloud storage.  You can learn the basics of automating report generation with Integromat in this brief video-tutorial:

 

 

  • ArcGIS Python API: The ArcGIS Python API is ideal if you want to generate reports in bulk in an automated way. For example, in case that you want to automatically generate reports for all surveys submitted daily or weekly. Refer to our ArcGIS Python API developer documentation for details.

Yeap! World UFO Day is approaching so we want to use this out-of-this-world release to remind you. This is a really substantial update to Survey123, including many, many fixes, new features and enhancements. We are really excited about 3.5 and we hope you will too! 

 

Survey123 Feature Reports

 

Many of you are already familiar with the Survey123 Feature Report capability, which we have had in Beta for quite some time. With this update, the Survey123 Feature Report service is no longer Beta. It is fully supported by Esri Technical Support and it is here to stay.

 

The Survey123 Feature Report service is designed to help you generate beautiful reports of your Survey123 records. You can define the contents and look and feel of your reports using a report template in Microsoft Word format. Once your template is uploaded to ArcGIS Online, the Survey123 Feature Report service can generate individual reports for records submitted from Survey123. 

 

The Feature Report functionality can be triggered from the Data tab of the Survey123 website as shown below. You can create reports for a single selected record, or in bulk, for a larger selection set.

 

 

You can control the layout, contents and look and feel of your output reports through report templates. One or more templates can be associated with a survey, so users in your organization with view access to the survey data can generate reports on their own.  Report templates are authored in Microsoft Word. You can add headers, footers, tables and format the text  in your template as you like.   Feature Report Templates include placeholders to bring responses from your survey record into the report. This includes text, dates, numbers, choices and checklists, photos, annotated images, signatures and of course, maps!  The syntax to bring data from your survey into a template is described in our Feature report templates help topic.  A full blog post dedicated to the feature report syntax is on its way.

 

Feature Report generation is available as a premium ArcGIS Online service. Users who want to use the service will need to be granted Feature report privileges. By default, all roles and role templates included in ArcGIS Online include this capability, but if you use custom roles you will need to turn it  on.  This can only be done by an administrator of an ArcGIS Online organization by including Feature report access to the custom member role. The next animation shows how:

 

 

As of today, and until the next release of ArcGIS Online (planned for October 2019), you can generate as many reports as you want for free. Starting October 2019, feature report generation will consume 2.5 credits per report file generated.

 

One final note for ArcGIS Enterprise users: The Survey123 Feature report service is designed to work and is fully supported with surveys and data hosted in ArcGIS Online. However you can also take advantage of this service with surveys hosted in ArcGIS Enterprise 10.5 or newer, with some limitations:

 

  • Your ArcGIS Enterprise instance must be accessible to the Survey123 ArcGIS Online Feature report service. That is, you must configure your  ArcGIS Enterprise web adaptor to expose access to your Enterprise instance from the internet. Please note that portals with Integrated Windows Authentication will not work with the report service.
  • The inclusion of any type of feature service attachments (photo, signature, annotated images...) is not supported when your data is hosted in ArcGIS Enterprise. This is a limitation listed in the backlog of ArcGIS Enterprise.

 

Bearing the limitations above, the use of the ArcGIS Online Feature Report service from your ArcGIS Enterprise instance will not incur into any ArcGIS Online credit costs.

 

 

Web surveys: Survey open, survey closed! 

 

Up until this update, if you wanted to disable access to a public survey,  you would simply unshare it. In this way, people could not longer submit data to your survey, but you could still view and analyze its results. The challenge with unsharing a survey is that when attempting to open your survey users would get a generic dialog asking for a user and password, and that is the last thing someone would expect.

 

You can now configure a "Survey closed" dialog and define when to show it. In the 'Survey closed" dialog you can add your own text, images and links describing to survey takers why survey responses are not being accepted. Through the Collaborate tab, you can also define the exact time when the survey will be closed or open.  For example, you can design and publish a survey today, schedule it to start accepting responses in November 1, 2019 and stop accepting responses in December 13, 2019.

 

 

This feature is currently only supported for web surveys.  The Survey123 mobile app will not honor the "Survey closed" custom dialog until a later release.

 

Web surveys: Avoid multiple submissions

 

This new feature is also built with web surveys in mind. The idea here is to prevent users from submitting a survey multiple times.  By default, users can submit data to your web survey as many times as they like, but you can now configure your survey to limit submissions to one. This is done through the Collaborate tab as well.

 

For clarity, the way this works is not bullet proof for public surveys.  If working with a public web survey, this setting will persist a cookie in the user's web browser. If the cookie is present, the user will not be able to submit again. A smart user can delete the cookie by clearing the browser's cache or running an incognito session in the browser.  Also, for public surveys, we can't really do anything if one person submits from different devices (a phone, a desktop computer in the office, a laptop at home).

 

 

If you really want to confidently block multiple submissions from the same user, the best option is to secure your survey to force users to identify themselves.  At that point, we can more consistently block multiple submissions.

 

Working with related records (repeats) in the Survey123 website Data view

 

Through the Data view in the Survey123 website you can interactively inspect all submitted records as a map, table and a form.  In this release, the table view and map views have been enhanced so you can navigate to related records in the table view and display or hide related records in the map. You can also apply filters to the table view.

 

 

In the example above, we have a Zika Household survey example. At each household, we have one or more household members and for each of them we document all trips they did in the past 4 weeks.  The member households and travel locations are modeled in the smart form as a nested repeat, and in the feature service relational model as two nested related tables. Look carefully at the table view and you will see that each of these layers/tables is shown in an independent tab. You can switch between the tabs, look at the data, apply filters and select records.  This gives you much more flexibility to explore your survey data!

 

Survey owner? Double-click to edit (Survey123 website)

 

One more enhancement on the Survey123 website Data tab. Double click on a cell in the table view... and voila, now you can edit the value.You can even edit related records. This function is only available to survey owners.

 

 

Photo Watermarking (Connect & Field App)

 

Over time we have received a number of requests to add watermarking capabilities into Survey123. This is important in some field data collection workflows; for example when performing certain damage ***assessments, documenting work performed or inspecting assets. 

 

The basic idea is to burn some critical information into the photo while it is taken from the device. Information such as the current date and time, the compass direction or the latitude and longitude. We have extended the XLSForm syntax so you can do all of this and much more.

 

In the example on the right we have a street service report survey. This is used to document work performed.  Note how the photo is watermarked automatically to include in the bottom-left corner of the image the unique identifier of the job. This job id comes from one of the questions in the form.

 

In the bottom-right corner you will see the date and time when the photo was taken as well.

 

You can control almost anything you want in these watermarks: the position, color, font, halo effects and of course the content, which can be dynamically calculated or retrieved from the form itself.  You can even use a logo in your watermark expressions!

 

To learn more about watermarks, check out our Survey123 Tricks of the Trade: Photo Watermarks blog post and our Watermarks help topic

 

Grid Theme (Connect & Field App)

 

Grid theme has been a long awaited feature, so we are really proud of announcing it today. Using grid theme you can control the layout of your survey to arrange multiple questions in a single row. This is important when you want to make your surveys resemble more closely paper forms, and when you want to take advantage of the wider screen of a tablet or desktop computer.  Using grid themes wisely can help you create much more usable forms to accelerate data capture because you can reduce the need to scroll up and down through your form.

 

Below is an example of a Certificate of Roadworthiness. No kidding, that word actually exists, you can google it .  Note how the form includes multiple sections: A first section at the top for the Tester's License Number, where we arranged a single question all the way to the right. Then a second section for the main title, with a single note centered in the screen. In section A (vehicle description) where we place questions, notes and even a signature pad side by side.

 

 

When taking advantage of grid themes, you will want to test thoroughly in your own field devices. A form with the layout shown above, for example, will work well on a tablet, but not on a smartphone.  Combining grid themes and multiple pages in your form is a really powerful combination too.  Grid themes and pages are worth their own blog post, so wait for one coming up soon. If you cannot wait, check our Appearance help topic for details on the XLSForm syntax, or the newly added sample in Survey123 Connect.

 

As of this release, grid themes are only supported in the field app, but we are working towards bringing them into the web as well.

 

Other miscellaneous topics (Connect & Field App)

 

You are going to find many other enhancements throughout Survey123 Connect and the field app in this release.  I simply want to highlight a handful, although you can find the complete list in our What's New help topic.

 

  • In Survey123 Connect, you can now preview your surveys in different form factors.

 

 

  • Constraint and required messages now appear inline, helping you identify the exact questions in your form that need attention. This is particularly useful when you have long forms, or forms with multiple  pages.

 

  • Many visual controls in the field app and Connect have been refreshed bringing a cleaner look.
  • The Survey123 field app now sends information about the type of submission (add or edit) to webhooks. This was introduced in our previous release for the Survey123 web app, and now we close the circle with this update for the field app.

 


Critical Fixes and Enhancements

 

There are a lot of new new features in this release, but in truth our biggest commitment for this update has been quality.  If you have been testing Survey123 through our Early Adopter Program you will recognize that many of the features we make official today had been on the works for quite some time. Our 3.5 release has been for the most part, devoted to fixing issues and applying some popular enhancements.

 

We want to thank all of you who have worked with us to validate many of these fixes over the past 10 weeks.  Here is a list of the most important bugs addressed.

 

Survey123 website and web app

 

  • BUG-000118801 An existing secured ArcGIS Server feature service added to ArcGIS Online with credentials saved is not be able to attach images when using the web browser to submit the survey (the data is submitted, but not the attachments).
  • BUG-000119623 Survey123, non GlobalID/Guid primary key field gets automatically populated with a GlobalID when submitted from the web form.
  • BUG-000120589 When a survey is published from Survey123 Connect, then modified in the Web Designer to check on "Allow the respondent to add another record" the option does not appear in the Web App.
  • BUG-000121051 Pulldata calculation on select_one question does not work when accessing the survey in a browser.
  • BUG-000122048 Survey123 doesn't submit survey through mobile browsers if the survey has pages and signature fields.
  • BUG-000121719 In the Survey123 for ArcGIS website, the total number of features displayed changes when logged in using the credentials of other people in the organization apart from the owner.
  • BUG-000122052 When running a report on a Survey123 for ArcGIS form that contains the Other option as a response from a Multiple Choice question, the Other option is repeated after each of the Other selected responses.
  • BUG-000122750 The sum() aggregate function does not concatenate string inputs and records a "NaN" value when submitting responses from a web browser.
  • BUG-000123131 Exporting results to a file geodatabase fails if a survey contains more than one nested repeat from the Survey123 for ArcGIS website.
  • BUG-000114635 Photos do not appear in the Analyze tab of the survey in survey123.arcgis.com when logged in as a custom role account.
  • BUG-000121812 The custom basemap is not visible in the Survey123 for ArcGIS web form when the basemap is not shared publicly.

 

Survey123 Connect and field app

 

  • BUG-000121623: Survey123 Connect for ArcGIS 3.3.51 and Survey123 Connect Classic 2.9.3 crash when using a pulldata() function that references a CSV with 1048,576 records.
  • BUG-000100441: Survey123 Connect: Truncated "Search" string and wrongly placed search icon in search box.
  • BUG-000103155 Surveys created from the Survey123 for ArcGIS Web interface that contain geopoint questions allow field users to zoom in further on the map than is allowable in the Survey123 for ArcGIS mobile app.
  • BUG-000103942 Large surveys crash the Survey123 for ArcGIS iOS field app when specific answers are provided on an iPhone 6 without saving data.
  • BUG-000113684 Surveys with multiple repeats with the repeat counts added crash upon opening in the Survey123 for ArcGIS (iOS) field app.
  • BUG-000113734 The survey creation performance degrades when greater than 5-6 repeats are added to a single survey form.
  • BUG-000114396 The Survey123 field app successfully submits the survey even when all the required questions in the Repeat field are not answered.
  • BUG-000097367: Truncation in Date field in Survey123 and Survey123 Connect.
  • BUG-000094992  Truncation in Publish survey in Survey123 Connect.
  • BUG-000094993: Truncated string in Settings-Style in Survey123 Connect.
  • BUG-000115227 When the language of an Android device is set to the Korean language, switching the device keyboard to Korean to type Korean characters into an autocomplete field in a Survey123 for ArcGIS survey is not possible.
  • BUG-000118014 Unable to use a Japanese keyboard in Survey 123 for ArcGIS (3.1.158) on Android devices. This behavior appears to be restricted to the Survey 123 for ArcGIS (3.1.158) release. This issue is not reproducible on iOS.
  • BUG-000118208 In the Survey123 field app, when a constraint is applied to a select_one question, it is not validated until submitting the survey.
  • BUG-000118366 Unable to send surveys in the Survey123 for ArcGIS field app when Layer ID has values other than the default value.
  • BUG-000119632 Once relevancy exposes a repeat in Survey123, the information in the repeat will get submitted regardless of whether the repeat becomes non-relevant again.
  • BUG-000120023 Select_one questions with an expression in Required or Relevant fields do not validate in Survey123.
  • BUG-000121057 In the Survey123 3.3 mobile app, a select_one question in a nested repeat conditional based on a select_one question within a single repeat will not display after changing the answer to the single repeat select_one.
  • BUG-000119414 In Survey123 Connect for ArcGIS, nested repeats are not preserved when scrolling through the Form Preview.
  • BUG-000121921 Data not displaying in first record of nested repeat with relevant expression referencing a question outside of the nested repeat when open from Drafts.
  • BUG-000122112 When required fields are not answered, the survey does not honor question ordering when alerting the user of required, unanswered questions.
  • BUG-000122534 Survey123 for ArcGIS hangs indefinitely when Form Item is moved or deleted from ArcGIS Online. An error message must be thrown instead.
  • BUG-000122255 The Survey123 for ArcGIS field app does not retain the font size setting on iOS and Android devices.
  • BUG-000121156 Submitting surveys from the Survey123 for ArcGIS field app retains related records from a repeat question, even after removing the attribute values for the same related records before submitting the survey.
  • ENH-000106588 Provide the ability to expand a Group in Survey123 by clicking the Group header text, not just the arrow.
  • ENH-000115797 Survey123 should allow users to have control over color schemes for the group box on a survey form.
  • ENH-000118020 Lessen the touch precision needed for the Group expand/collapse button in the Survey123 field app.
  • ENH-000120369 Clarify the messaging when deleting a survey from Survey123 Connect to be more explicit that AGOL/Portal items will be deleted.
  • ENH-000121453 Provide additional functionality for there to be two "Text" question types be adjacent on one line in the "Form Preview" page of the Survey123 Connect for ArcGIS, instead of only one "Text" question type on top of another "Text" question type.


Next steps

 

We are now furiously working on 3.6. Join our Survey123 Early Adopter Program to test some of the new features we are working on, including:

 

  • Support for custom webmaps and mobile map packages.
  • On-premises setup for the Survey123 website and Survey123 REST API.
  • Support for polygons and polylines.
  • And more...

 

For developers customizing or branding the Survey123 field app, we will be making version 3.5 available with AppStudio for ArcGIS 4.0 Beta in the next weeks.

 

Do you believe?

Here we are once again, so close to the Esri User Conference!  Here is a short guide to help you find Survey123 related events at the conference.

 

Technical Sessions:

 

Technical Sessions are one UC's favorites. Always well attended, Tech Sessions run for about 60 minutes and cover technical aspects of Esri products.  We always try to keep a good chunk of time at the end for a live question & answer section. Tech Sessions start on Tuesday morning and run until Friday.  They all take place in the second floor of the San Diego Convention Center (also known as SDCC).

 

If you want to start planning your sessions, go to our UC 2019 Online Detailed Agenda. You can search sessions by topic and presenter and if you are logged-in you can add sessions to your agenda.

 

Here is a list of some of the Survey123-related Technical Sessions we are offering:

 

  • Survey123 for ArcGIS, an introduction: This session covers the basics of the Survey123 website, including designing smart forms with the web designer, publishing and sharing surveys, generating high quality reports for submitted data and other capabilities available through the Survey123 website. The session will end with a sneak peak of smart form capabilities going beyond what is possible through web designer.  I recommend attending this session even if you are already familiar with Survey123, because there are many new things that have been added to the Survey123 website in the past few months.  Philip Wilson, James Tedrick and Zheng Liu and will be leading this session; fun and good content guaranteed!
    • Tuesday, July 9 10:00 am - 11:00 am SDCC - Ballroom 08
    • Wednesday, July 10 2:30 pm - 3:30 pm SDCC - Ballroom 06 E
  • Survey123 for ArcGIS: Working with webhooks: Learn how to use web hooks with Survey123 to automate all sort of tasks aftere a survey is submitted. Automate e-mail notifications, instantly move Survey123 data into databases and spreadsheets, integrate Survey123 with image recognition software and much more.  Ismael Chivite, Brandon Armstrong and Calvin Jung will show you the way!
    • Wednesday, July 10 10:00 am - 11:00 am SDCC - Ballroom 09
  • Survey123 for ArcGIS: Advanced Topics: This session will introduce advanced capabilities of Survey123 including the new theme-grid layouts, editing existing features and designing surveys on top of existing feature layers, workflow automation with Survey123 and web hooks and other exciting features. There will be lots of live demos and time for live questions.  Ismael Chivite, Elvin Slavik and James Tedrick will present this session.
    • Thursday, July 11 1:00 pm - 2:00 pm SDCC - Room 10
  • Survey123 for ArcGIS: Extending the mobile app: Awaken the developer in you. Join this session to learn how you can extend the Survey123 mobile app with your own Javascript functions, the new extensibility framework or even brand and create your entirely new Survey123 mobile app. Presenters: Ismael Chivite, Elvin Slavik and Erwin Soekianto.
    • Thursday, July 11 2:30 pm - 3:30 pm SDCC - Room 31 B/C
  • ArcGIS Apps for the Field: An Introduction: In this session we will describe how all ArcGIS mobile apps work together. A great place to learn how Survey123 works along with Workforce for ArcGIS, Navigator for ArcGIS, Collector and even Operations Dashboard for ArcGIS.  Scheduled presenters include Scott Ball and James Tedrick. 
    • Tuesday, July 9 4:00pm to 5:00pm SDCC - Room 2
    • Wednesday, July 10 8:30am to 9:30am SDCC - Room 2

 

Survey123 User Group Meeting

 

We are happy to announce that this year we will have our own Survey123 User Group Meeting. It runs on Tuesday at lunch time from 11:30 AM to 12:30 PM, but no lunch is provided. This is for the ones with a strong Survey123 spirit!  Our User Group will be a great opportunity to meet other Survey123 Ninjas. This is fundamentally about connecting with other people and the team.  We will have a couple of 7-minute lighting talks to warm-up and then the rest of the time is for group discussion and networking.

Survey123 User Group Meeting: Tuesday 9,  11:30 AM - 12:30 PM  SDCC - Room 8

Demo Theaters

 

You can think of Demo Theaters as mini-Tech Sessions. They are shorter, always focused on very specific workflows and run in a small setting were questions can be handled throughout the presentation.  The Demo Theaters take place in Expo area (first floor), close to the Product Islands.

 

 

User Presentations

 

User Presentations are a great way to learn the details of how Survey123 is applied to specific workflows. A good opportunity to learn directly from users like you!

 

  • Mobile data collection with Survey123: There will be two user presentations highlighting how Survey123 is being used with Spike Laser Measurement devices and with high accuracy GNSS receivers to streamline field data collection workflows.
    • Wednesday, July 10 2:30pm - 3:30pm SDCC - Room 25C
  • Going mobile for the Homeless Point in Time count: Paper presentations from multiple users sharing how they used Survey123 for ArcGIS to support the Homeless PIT count.
    • Wednesday, July 10 1:00pm - 2:00pm  SDCC - Room 29A
  • Creating impacful applications for Education: This session presents two case studies of impactful GIS applications, including employing Dashboard for ArcGIS to improve earthquake safety and preparedness in K-12 Schools; using Survey123 for real time data collection, environmental resource protection, and sanitation management. The case studies in this session exemplify the best practices for creating impactful GIS applications
    • Tuesday, July 9 1:00 pm to 2:00pm SDCC - Room 25C
  • Crowdsourcing water quality data collection with Survey123: By the Arizona Department of Environmental Quality. With more than 45,000 miles of stream and river beds in the state, this is far too much for agency to staff visit. Utilizing the tools of Survey123, three forms were created to have citizen scientists help in collecting the data needed to protect this valuable resource.
    • Wednesday, July 10 8:30 am - 9:30 am SDCC - Room 29C
  • Innovation of ArcGIS Online Apps to Enhance British Military Working Practices: The presentation will demonstrate  innovation and implementation of the ArcGIS online applications used by HQ 1 (UK) Division Geospatial Support Cell. Applications have provided the British military with a fantastic opportunity to dramatically increase their operational effectiveness.
        Wednesday, July 10 1:00 pm - 2:00pm  SDCC - Room 26A
     

 

Expo

 

The Expo area is opens Tuesday and closes on Thursday afternoon. It is located in the first floor in the San Diego Convention Center (SDCC) and features different areas including Partner Exhibits, Esri Industry Neighborhoods, GIS Product Areas and Customer Care booths.

 

  • Product Areas (Islands) at the Expo:  You will always find someone from the team at the Survey123 booth. This is the place to go when you want to have a one on one conversation to address specific ideas or issues you may have. We often refer to the Product Areas as Islands. The Survey123 team will have a station under a big sign that says 'Apps'. It will be a big sign... you will see it.
  • Partner & Sponsor Exhibits: The Survey123 software is only a third of the story. There are two good reasons to visit the Esri Partner and Sponsor Exhibits. The first one is that many of them are now experts building smart forms with Survey123 and know the tricks of the trade when implementing solutions. Swing by their booth and ask them about their own experience.  The second one is that through the Partner Exhibits you will find many hardware vendors that can teach you the benefits of using their offerings with Survey123. Think Laser Measurement Solutions, GNSS receivers, ruggerized devices and much more. Just to name a few:
  • Customer Care and Esri Tech Support: Our Esri Customer Care and Technical Support teams will be represented in the Expo. You can also swing by their stands with your Survey123 technical questions.
  • Esri Industry Neighborhoods: Everything from Public Safety to Defense, Local Government, Utilities and more.  The Industry Neighborhoods are a good place when you want to connect with people that understands how to put Survey123 in the context of your own workflows.  Ask for a demonstration of Survey123 focused on your problem or for examples of organizations using Survey123 in your industry.

 

Esri Education Summit

 

The Esri Education Summit takes place between July 7-10 at the Marriott Marquis & Marina Hotel. You will need to register separately for this one!

 


 

Pre-Conference Seminars

 

 

By now, many Pre-Conference Seminars are sold-out, but I thought I would include info about this for those of you who are planning to come next year. You need to hurry before all seats are  taken.  Survey123 will be highlighted in the Field Operations and ArcGIS PreConference seminar:

 

  • Mastering Smart Form Design with Survey123 and XLSForm
    • In this one-day hands-on seminar you will learn how to use the XLSForm specification to build smart forms with Survey123 for ArcGIS. We will introduce the XLSForm specification and will dive into advanced techniques to implement custom data business validation rules, conditional statements to control the visibility of portions of your form, auto-calculation rules for attributes in your features and much more.
      • Sunday, July 07 8:30 am - 5:00 pm
  • Field Operations and ArcGIS
    • Learn how ArcGIS helps your organization optimize field operations by: taking ArcGIS maps offline into smart devices using ArcGIS Publisher Extension and Explorer for ArcGIS; coordinating your field workers with Workforce for ArcGIS and Navigator for ArcGIS; improving locational awareness by knowing the location of key staff and equipment with Tracker for ArcGIS; and optimizing data capture with Collector and Survey123 for ArcGIS; building effective operation dashboards with Operations Dashboard for ArcGIS.
      • Sunday, July 07 8:30 am - 5:00 pm

 

The Esri Events App:

 

If you are coming to San Diego, I recommend that you download the Esri Events app. It  will help you follow your own schedule, keep track of last minute changes to sessions and vote sessions too!

 

 

If you are coming this year, welcome!  If not, be ready for next. Have a look at the many reasons why you should not miss this event!

In this blog post I will explore a few techniques to take your use of Survey123 within Integromat to the next level.  I am going to assume that you already know the basics of working with Survey123 and Integromat. If not, you may want to start with Getting Started with Survey123 and Integromat

 

Many more posts like this could be written, because what you can do with Survey123 and Integromat deserves an entire book, but we need to start somewhere. So here are a handful of ideas. I hope you discover some new things:

 

Working with Survey123 attachments

 

I will start with this one because the topic of working with photo attachments from Survey123 in Integromat has come up many times before, and it is not entirely obvious how to deal with them.  Say your Survey123 smart form includes a photo and a signature question. Can you work with them in Integromat? Can you include the photo and signature images as attachments in an e-mail? Can you back-up the images in OneDrive or Box? Yes to all of that and here is how.

 

The trick is using the Integromat HTTP module, which includes an action called 'Get a file'.  This action takes the URL of an image as an input, which you can get from the payload of a Survey123 record.

 

  1. Feed the Survey123 'watch survey' module with a smart form that includes one or more photo/signature questions.
  2. Use the Get a file action in the HTTP module to download your attachment.

 

 

The payload of 'Watch a survey' includes a collection of attachmentInfo objects. They are uniquely identified with the name of the question generating the attachment and include a url property that can you use to invoke 'Get a file'.

 

Once you have the file, you can do a handful of things with it. Of course, you can now include it as an attachment to an email.... That is a classic one!

 

 

But you can do more interesting things too. For example:

 

  • Back-up your photos and attachments in Google Drive, Microsoft OneDrive, DropBox.
  • Upload images to Flickr, Instagram, Twitter or other social media.
  • Reduce the size of an image before storing it.
  • Extract metadata from the photo, including its name, size or format.
  • Process the photo with Google Cloud Vision to identify elements within the photo or simply extract text from it.

 

 

 

Automating Survey123 custom reports

 

The Survey123 module includes an action to help you automate Survey123 feature reports. Not familiar with reports in Survey123? We have a video for that here and an introductory blog post too. The Create Feature Report action is pretty much self-explanatory: it takes an input survey, a report template and the objectId of the survey record you want to include in the report.  If the trigger of your scenario is a 'Watch a survey' trigger, then you can use again the payload of your Survey123 web hook to extract the objectId. In this case, it is within the Feature-Result object as shown in the next animation.

 

 

The use of the HTTP module to download files as described above also applies to the outputs of the Survey123 report action. That is, after you create a custom report file, you can download it and include it as an attachment in an email, or upload it to cloud storage or FTP site.   Since many cloud storage solutions include options to automatically synchronize the contents of local and cloud folders, I like to upload the Survey123 report files I create to the cloud so they show up in my computer right away. I know, I am old fashioned, but that feels like magic.

 

When configuring the Create Feature Report action you will notice that at the bottom of the dialog you can expand a window with advanced settings. In there, you will be able to configure the UTC-Offset. This setting will be your friend if you plan to include dates and times in your reports.  Keep in mind that all data within Survey123 -and ArcGIS- is stored in UTC time. Since Integromat has no real context as from where in the world the report is being invoked, you will need to adjust the UTC offset yourself.

 

The Survey123 Custom Report service is currently in Beta and scheduled to be generally available in version 3.5

 

Back to the HTTP module

 

Earlier, we looked at the 'Get File' action in the HTTP module. This time we will revisit the HTTP module but this time to describe its 'Make a request' action. I personally find this action extremely useful, because it allows your scenarios to interact with pretty much any external web service you want. Using the 'Make a request' action you can for example invoke the ArcGIS REST API to update an attribute in an existing feature, or to simply query a feature service. Lets get down to very specific scenarios:

 

  • Make a point- in-polygon query to an ArcGIS feature service to get the postal code of the submitted Survey123 record location. Then make another request to add the postal code value as an attribute of the submitted feature.
  • Use information from a submitted Survey123 record to automatically create a new assignment in a Workforce for ArcGIS project (make a request to add a new feature into the Workforce assignments feature layer).
  • After having invoked Google Cloud Vision to categorize the contents of a photo submitted via Survey123, update the attributes of the Survey123 feature accordingly.
  • Update an image_url attribute in the submitted survey so you can later create a popup in ArcGIS that shows your image directly when the popup is open.

 

 

The mechanics for using the 'Make a request' action imply good knowledge of the target web service your are invoking. I will use next a simple scenario to illustrate the essence of this technique. Say for example we want first to get the location of a submitted survey record, get the current temperature at that location using a weather service and store that temperature value as a GIS attribute of the feature that was originally submitted.

 

Here is how you get the current temperature. OK: so far we are not using 'Make a request' anywhere. We are just setting this up.

 

 

Next, we are going to 'Make a request' to the survey feature layer using the ArcGIS REST API to add the temperature to the temperature field of the submitted record. Specifically, we are going to make a request to the Update Features REST API operation. This operation, as described in the Updates Features ArcGIS REST API Developer documentation, is located at:

https://<featurelayer-url>/updateFeatures

All information needed to construct the URL above is included in the survey web hook payload. It is a matter of combining the surveyInfo.serviceUrl and the feature.layerInfo.id properties, followed by updateFeatures.

 

 

Again according to the doc, this operation only supports POST requests and the encoded parameters include:

 

  • f to define the output format as json or html. This is optional but handy so we can debug, if needed, the output within Integromat.
  • features to define the actual updates we want to make in the layer. The content of this parameter is a json object with information about the records and fields we want to change.
  • token to specify a valid token we can use to securely access the feature layer. This parameter is not needed if the feature layer is shared publicly.

 

All of that, translates into Integromat as follows:

 

 

The most error prone step is adding the exact JSON value for the features parameter. I like to copy the sample JSON that comes in the developer documentation, and then modify it for my own purposes.  It is very important to note that the sample JSON in the documentation must be enclosed with extra curly brackets. In my case, this is the JSON string that will go in the features parameter. Note that the objectId in green, and the temperature in brown are dynamically populated using outputs from other modules in the scenario.

 

{

    "attributes" : {
      "OBJECTID" : objectId,
      "Temperature" : "temperature"
    }

}

 

You will see yourself going through some trial and error until you get it right, but since you can easily look at the output of the 'Make a request' operation, it typically does not take very long before you figure things out.

 

Using the 'Make a request' action you can do a lot of things. Updating attributes is one of them, but you can really do many others.

 

A nice complement to the 'Make a request' action in the HTTP app are the many out of the box tools included in Integromat to parse XML, JSON, HTML and the many functions to handle text, dates, arrays and much more. They will help tremendously when you need parse responses from the web services you are invoking.

 

Handling Errors

 

This is all getting very interesting, but as you create more sophisticated scenarios you will face situations where you need to handle run-time errors.  If you are dealing with attachments, what happens if the end-user does not submit one? Or what happens if you hit an external web service and you get a bad output, or a time-out?

 

By default, when an unexpected error occurs, the execution of the Integromat scenario is stopped at that point but you can handle errors in other ways. You can for example ignore the error and continue executing, or you can choose to branch off the execution to do different things. This is all described in the Introduction to error handling tutorial in the Integromat Help Center.

 

The following animation shows how you can add an error handler in case the 'Get file' action fails downloading a non-existing ArcGIS attachment. In this case I chose to simply invoke the Gmail connector to send an e-mail without attachments. If an ArcGIS attachment is found, it gets added to an e-mail, otherwise the email is sent without attachment.

 

 

To learn more about error handling in Integromat, I recommend reading the Advanced error handling help topic.

 

Connecting Integromat to an ArcGIS Enterprise instance

 

By default, the Survey123 modules in Integromat will connect to ArcGIS Online, but you can actually make Integromat work against your own instance of ArcGIS Enterprise.

 

Before we start, it is important to highlight that Integromat needs to communicate with your own ArcGIS Enterprise instance, and this will not be possible if ArcGIS Enterprise is running completely behind your firewall.  You need to make your ArcGIS Enterprise instance is available to Integromat over the internet. ArcGIS Enterprise administrators are familiar with this concept, which is often resolved by exposing ArcGIS Enterprise through a reverse proxy in the DMZ, although there are other options.

 

A second step, involves explicitly registering Integromat as an app in your ArcGIS Enterprise instance. This step is necessary for Integromat to be able to securely access information in your ArcGIS Enterprise. Follow these steps:

 

  • Log into your portal and go to Content.
  • Add a new Item of type Application.
    • Type: Application
    • Title: Integromat (Really, you can call this anything you want)
    • Tags: Survey123 (Same here, any tag you want)

 

 

 

 

All the above is necessary for Integromat to be trusted before you can initiate a secure connection to your Enterprise instance.

 

Now, it is time to create a new Connection within Integromat to your ArcGIS Enterprise instance.  While adding a new web hook in the Watch a survey or Create a report modules, choose to create a New Connection:

 

  1. In the Create a New Connection dialog, expand the advanced settings.
  2. Add the https URL of your ArcGIS Enterprise instance, including the web adaptor name.
  3. Populate the AppID and AppSecret properties using the App Registration information from the Application item you created before in your portal.
  4. Click on continue to authenticate against your ArcGIS Enterprise instance.

 

 

You will be able to re-use this connection again and again when creating new web hooks.

 

Add and Edit Trigger Events

 

Initially, we added support for Add trigger events, meaning that the Survey123 web and field apps would be able to trigger a web hook when a new record was submitted from a smart form.  We are progressively adding support for edit trigger events as well, so you can launch your Integromat scenarios when a record is updated too.

 

Support for edit trigger events was added in the Survey123 web app in our Comet Halley Update (May 23, 2019). The Survey123 field app will catch up starting with version 3.5 (scheduled  to be released on June 27, 2019).

 

If you do not now how to update records using the Survey123 web app, check the Survey123 Tricks of the Trade: Editing records in a web form blog post.

 

While you can configure your Integromat scenarios to be triggered by Add and Edit events, please be aware that the Survey123 field app will not honor this setting until version 3.5. As of 3.4, only the Survey123 web app supports web hook edit triggers.

 

Below, you can see how you can configure Integromat to define which triggers you want to activate in your web hook.

 

 

If you happen to activate both of them, you can use the output of the Watch survey module output to tell which event triggered your scenario. This is handy in case you want to build a single scenario and add branching logic to handle adds and edits differently.

 

The trick here is that the eventType property is only included in the payload once there is an actual submission of data. You will actually not see the eventType property pre-populated in the output, until you send data.  When working with eventType, I typically put the scenario in listening mode (Run once) and submit data to populate the property. This helps me build the rest of the scenario more easily. Once the scenario is running for good, this property will change accordingly.

 

 

 

Blueprints

 

Did you know you can export your scenarios as a JSON file and import them later?

 

 

I included as a simple blueprint of the weather sample as an attachment to this blog so you can play with it. All you need to make it work is a survey with a geopoint question and an additional field where you can store the temperature. Make sure you adjust the name of your temperature field in the features parameter of the 'Make a request' module. By now I am guessing you do not need that I create another gif. I am completely "giffed-out" for today.

 

As I said at the very beginning, there is so much to write about Survey123 and Integromat, so this is just a starting point. If you feel there is a particular topic of your interest that should be covered in more detail, do not hesitate asking. Now,  let me anticipate that we still have not figured out just yet how to work comfortably with repeat data within Integromat.

 

Despite the repeat debacle, I hope this blog post will give you some ideas to have more fun with Survey123 and Integromat. I also wish some of you will share some of your creations. Share a screenshot of your scenario and a brief description of what you accomplished with it, or even better the blueprint.  Add them as comments below to inspire others!

Integromat is a powerful tool for automating manual processes. With Integromat, you can easily connect Survey123 with many other apps and services. For example, you can configure a scenario in Integromat to perform an action immediately after data is submitted or edited via a Survey123 smart form: send an e-mail or an sms, copy the data into a spreadsheet or database, create an assignment in Workforce for ArcGIS or simply perform a reverse geocode to associate that address with the record that was just submitted.

 

Integromat includes well over 300 different out of the box 'integrations' (and growing), which you can combine visually in scenarios to automate specific tasks. You will find integrations for everything ranging from Office365 to Box, PayPal, SQLServer, Google Calendar, Apple Push Notifications, InvoiceNinja, Instagram, Google Cloud Print, and of course, Survey123 for ArcGIS!

 

 

The basics, in four minutes

 

To get started with Survey123 and Integromat I strongly suggest you watch this four minute video-tutorial. It will give you a good overview of the whole thing, plus enough detail to follow along and build your first Integromat scenario.

 

 

You can create a free Integromat account to test the waters. Note that for production use you may want to upgrade to a paid subscription plan. There are multiple options, with increasing levels of support, operations, data transfer, etc.

 

Get a kick-start with a template

 

Also for starters, you can go with the different templates we have created. With them, you can automate common tasks with minimum effort. If you find a template that fits what  you want to do, you can create a scenario out of it with a simple click. Integromat will guide you, step by step, so you can configure the template with your own Survey123 smart forms and other apps as appropriate.

 

 

Integromat Survey123 Templates

 

Once you bring a template into your own scenario, you can also extend it to do even more...

 

One hour for more details

 

Derek Law and Brandon Armstrong recently delivered a free Live Training Seminar focused on automating e-mail notifications.  This seminar was recorded and features Integromat. This is good material as well if you are getting started.  You will need to sign in with your Esri credentials to access the recording.

 

Automating e-mail notifications with Survey123 Live Training Seminar

 

Integromat Help Center

 

To get the best of Survey123 and Integromat you need to know them both. As you work with Integromat and Survey123 you will realize soon that this is more than just about sending an e-mail notification when survey data is submitted.   Integromat is actually a very powerful visual environment where you can model common business workflows. Integromat is also more than 300 out of the box integrations. It includes quite flexible options to control the flow of data and decisions within a scenario, with iterators, routers, filters and error handlers.  You can also configure your scenarios to execute at regular intervals and perform operations in bulk. 

 

What I like the most about Integromat is that it makes it possible to do lots of things for which in the past you would need programming skills. This makes more people, more productive. Having said this, if you have developer skills you will also be impressed with all the things you can do.  The more  you know, the better.

 

Spend some time roaming around the Integromat Help Center. Check out their video-tutorials and technical articles to learn new techniques. It will pay off!

 

Once you get on the basics, check out theSurvey123 Tricks of the Trade: Integromat blog post for additional info!

We do not seem to observe the stars as much as we used to, but it is fascinating to look up and think how much our life as a species has depended on a good understanding of the sky. We have dedicated this update to the comet Halley. Tomorrow, over 2,250 years ago, the first documented sighting of comet Halley was made by Chinese astronomers. It was sighted May 24, 240 B.C.  There are potentially earlier accounts from 467 B.C when Chinese and Greek astronomers logged the sighting of a comet, which could also have been comet Halley. We will go with the more certain date of May 24, 240 B.C and celebrate it along with this update of Survey123.

 

The passing of comet Halley has been very well documented since these initial accounts. Initially there was no evidence that all these sightings were of the same comet. The comet's periodicity was first determined in 1705 by English astronomer Edmond Halley, after whom it is now named. It visits us regularly every 75 years. It came last in 1983 and will return in July 2061.

 

This update is focused exclusively on the Survey123 website. Clear your browser cache!

 

Here is what is new:

 

Enhanced Integromat support

 

Integromat is an amazing online automation platform. With it you can easily automate tasks, visually connecting all sorts of apps together. We strongly believe that facilitating your work with platforms like Integromat can tremendously help you embed Survey123 within larger enterprise workflows, making sure your Survey123 data flows nicely into the way people work within your organization.

 

Many of you have been experimenting with Integromat and Survey123 since we first added support for webhooks in 2018. The uptake of webhooks with Survey123 has been pretty spectacular!  Some of the most common task automations we have seen include:

 

  • Automatically send an email or SMS when an inspection or incident report is submitted from Survey123.
  • Instantly add a new row into an Office 365 Excel spreadhseet when data is submitted from Survey123
  • Create a Workforce for ArcGIS assignment and a Google Calendar event when data is submitted from Survey123
  • Automatically reverse geocode the location a submitted survey and store the address in a GIS attribute of the submitted survey

 

We have worked closely with our friends at Integromat to add a new Survey123 Integromat module. This means that you can now bring your Survey123 data into Integromat much more easily and do much more with it than before. Jim Moore put together this brief but informative video to give you a quick overview of this new Survey123 Integromat module. Here it is:

 

 

In our Getting Started with Survey123 and Integromat blog post you will find a brief guide with resources to learn  the basics of working with Survey123 and Integromat.

 

For those of you already familiar with Survey123 and Integromat, you will find that this new module, lets you do some more advanced things you have been asking for.  For example:

 

  • Easily work with photo, signature and markup attachments. Include them as attachments in emails, upload them to Box, Microsoft OneDrive or Google Drive.
  • Automate report generation through the Survey123 Feature Report service. For example, for every survey submitted, automatically trigger a new  report using a custom template and then email or update that final report.
  • Trigger Integromat scenarios when Survey123 is used to update existing records*. Add a router in your Integromat scenario to do certain things if a record is submitted, and other things if the record has been updated.
  • Setup an Integromat scenario on top of ArcGIS Enterprise.

 

All of the above are covered in the Survey123 Tricks of the Trade: Integromat blog post.

 

* Please note that the field app will not trigger web hooks on edit events until our upcoming 3.5 update.

 

Delete and update submitted records (Survey123 website)

 

You can now use the Data tab in the Survey123 website to delete and update submitted records. This comes very handy if you need to do some quick cleanup of test data during your survey design process, or to make adjustments to already submitted data to your survey.

 

This is how it works: First login to the Survey123 web site, select a record in the table shown in the Data tab and then use the Edit button in the panel on the right to enable editing on the individual response form. Make your changes and submit.  The following animation shows the key steps:

 

 

You can use the Edit and Delete options for both surveys created in Survey123 Designer as well as Survey123 Connect. However, if your surveys were created in Survey123 Connect, be aware that the Edit button will be disabled when working against a survey with a repeat.

 

Update submitted records from a Survey123 web form

 

You may be already aware that you can update existing records in your feature layers using the Survey123 field app. Starting with this update you can update existing records with Survey123 from your web browser too! This is possible by opening your web form with URL parameters.  TheSurvey123 Tricks of the Trade: Web form URL parameters blog post introduces the concept of web form URL parameters. Now you can use URL parameters to edit existing records.

 

The syntax is fairly straight-forward:

 

  • Set the mode to edit: mode=edit
  • Pass the objectId of the record you want to update: objectId=13 for example.

 

For example:

https://survey123.arcgis.com/share/815b69a16e2f474e93da7e2eb0255e5d?mode=edit&objectId=13

 

Editing is not supported in web forms published prior to this update. If you want to edit records with surveys you published in the past you have two options:

 

    1) Dynamically upgrade your survey at run-time by adding the version=latest URL parameter

    2) Republish your survey to force a version upgrade at publishing time.

 

The key to all of this is to somehow dynamically construct the URL with the objectId of the record you want to update. In the Survey123 Tricks of the Trade: Editing records in a web form blog post this is all discussed in more detail.

 

Other enhancements and fixes

in the Survey123 website and Survey123 web app

 

 

  • Save Survey As... now works also against surveys that you do not own. 

 

This is very handy if you want to clone survey designs within your ArcGIS organization. Create a survey first using Survey123 Designer, then share the survey with someone else by granting access through the Viewer group sharing settings in the Collaborate page. Next, have that second person login into the Survey123 website and show 'Surveys I can view results for' in the survey gallery. The Save As option will be available on the survey tile menu to clone the survey. 

 

 

The Save As... option in the Survey123 website is only available for surveys authored in Survey123 Designer. If you want to clone a survey authored in Survey123 Connect, use the Save As... option in Survey123 Connect

 

  • You can now quickly switch between ArcGIS accounts in the Survey123 website without having to logout.

 

  • An option to optionally propagate selection filters between the Data and Analyze pages has been added.

 

Filter Options in Data Page

 

  • BUG-000113962 Unable to type certain Japanese character in the hints and explanations of headers in the Survey123 for ArcGIS Web Designer.
  • BUG-000120077 Unable to export data from Survey123 website when logged in as a member having 'User' Role and ‘Creator’ User Type.
  • BUG-000120887 When opening an existing Survey123 for ArcGIS form from ArcGIS Online using the 'Manage in Survey123 Website' option, the error message, "Your account () is not licensed for Survey123. Please ask your organization administrator to assign you a user type that includes Field/Essential Apps..." is returned.
  • BUG-000120055 Reports may fail to generate from a survey response that contains a repeat with many high resolution images.
  • BUG-000120222 Unable to generate a Survey123 for ArcGIS Report if there is an IFstatement containing a choice with an apostrophe.
  • BUG-000120851 Survey123 for ArcGIS web app displays the World Geocoding Service (WGS) on geopoint questions, despite removing WGS from the organization's Utility services.
  • ENH-000115658 Allow for the functionality to configure right to left labels and choices in Survey123 forms.
  • BUG-000113592 Questions label disappear from the survey when default language is changed from English to French or Spanish in a browser.
  • BUG-000112106 Survey123 for ArcGIS web app does not respect the language settings of date type questions in the browser.
  • BUG-000114761 Field Matching and string-length(.) constraints cause a green "It's an invalid answer." subtext to persist in Survey123 for ArcGIS web site under the questions that are constrained that does not disappear after fulfilling the constraints or leaving the fields blank.
  • BUG-000115247 Survey123 for ArcGIS web application does not respect the default language settings in the browser.
  • BUG-000117184 Submitting an Image through the Image Upload Question in Survey123 through the Web Designer in a Desktop Browser fails if the Survey was created with the Web Designer
  • BUG-000116320 The Survey123 for ArcGIS web app displays Image and Geopoint questions in English even if the web browser is set to another language in Internet Explorer.
  • BUG-000120426 Conditional calculations do not populate as expected for select_one questions in the Survey123 for ArcGIS web app.
  • BUG-000118413 Reverse geocoding results are different between Survey123 Connect and Survey123 web application.
  • BUG-000120387 Survey123 web app fails to record time when a time between 12PM and 1PM is selected
  • BUG-000121290 Loading a survey in a web browser with a Toolbar Background Color assigned causes the default color scheme to show momentarily before switching to the chosen color.
  • BUG-000120589 When editing surveys published in Connect the prompt to save changes on exit is not necessary if changes were published, does not work if changes were not published.

 

 

About the next update

 

 

As mentioned above, this update is exclusive to the Survey123 website. As usual we wanted to release both updates for the website and Survey123 field app together, but we decided to pull back on the Survey123 field app at this time. We are doing our best to bring some important fixes you have reported to the Survey123 field app soon. The current schedule is set for an update in 5 weeks (late June 2019). Our Survey123 for ArcGIS Early Adopter Program is making early beta builds of our next update available for testing.  The Survey123 website will also be updated in late June.

Today we will explore how you can edit existing records using the Survey123 web app, also known as a web form. The idea is quite simple, we will populate the contents of a Survey123 web form using an existing record from a feature layer and let users update (edit) that information. When working this way, the submit button in the survey will update an existing record, as opposed to creating a new record for a new survey response.

 

This technique for editing data in a web form be quite handy. You can use it for example to let people review and modify existing submitted data, or to simply let them finish an uncompleted form. You can also incorporate web forms into QA/QC and data validation workflows.

 

Before we start, be aware that the editing of records in web forms is only available in the Survey123 web app after May 23, 2019 (version 3.4 and newer). I will describe later in this blog how you can make your surveys run in this latest version, even if you published it in the past in an earlier version.

 

The Basics

 

Editing of existing data in a web form is possible through the use of web form URL parameters. If you are not familiar with web form URL parameters in general, I strongly suggest you read the Survey123 Tricks of the Trade: Web form URL parameters first.

 

Here is an example:

https://survey123.arcgis.com/share/1cb28b212b5542acbbdbaa35feba0765?mode=edit&objectId=9

Note the two URL parameters:

 

  • objectId=9, this parameter loads the record in your feature layer with objectId 9 into your form. Of course, you can replace 9 with the specific objectId of the feature you want to load.
  • mode=edit, puts the web form in edit mode. This essentially ensures that when the user hits the submit button, the current record is updated. If the mode is not set to edit, the form will create a new record when submitted, even if the form was initialized with data from a particular record.

 

URL parameters are case sensitive. For example: objectId is not the same as ObjectID or objectID. If you see that your form is not loading the record you want, double check the spelling of the parameter, and also ensure the objectId you are trying to load exists.

 

As described above, the editing mode in web forms was incorporated in version 3.4. If you want to edit records using a survey that was published in the past, you can bring things up to speed in two different ways:

 

  • Upgrade your survey by simply re-publishing it. If you originally published your survey using Survey123 Web Designer, then use Designer to publish it again. You do not need to make any changes other than simply clicking the Publish button. The same goes with Survey123 Connect for ArcGIS: If you originally published your survey with Connect, go back to Connect and hit publish.
  • Upgrade your survey at run-time: You can alternatively add the version=latest URL parameter to force the survey to be loaded using the latest available Survey123 web app. Something like this:

 

https://survey123.arcgis.com/share/1cb28b212b5542acbbdbaa35feba0765?mode=edit&objectId=9&version=latest

 

My recommendation is that whenever possible you upgrade your surveys by re-publishing them. It will improve the initial loading time of the survey in the web browser, and it will guarantee that the behavior of your web form remains unchanged even if the Survey123 release changes.

 

From a practical perspective, the technique described above is useless unless you somehow present a ready-to-use link (URL) to the end user; A link that already includes the appropriate objectId to edit. Do not expect end users to modify that objectId manually in the web browser URL address!  I will next describe three scenarios to illustrate how you could apply this technique.


Launching an edit web form from a feature popup

 

Our first scenario allows users to navigate a map, and launch an edit web form from the popup of an existing record. The key is to customize the feature layer popup so it dynamically generates the correct URL to launch the survey. Once the popup is configured, you can use the web map within the ArcGIS Map Viewer, custom Web AppBuilder apps, Explorer for ArcGIS and any app that supports web maps.

 

Here is a brief step by step guide to configure your popups:

 

  1. Using the ArcGIS map viewer, add your survey feature layer (or hosted feature layer view) into a web map.
  2. Open the configure popup dialog for your survey feature layer.
  3. In the popup contents section, select a custom attribute display.
  4. Add the URL of your web form into the attribute display. Then add the mode and objectId parameters.
  5. Dynamically populate the objectId parameter using the OBJECTID field from your feature layer.
  6. Use the full web form URL with its parameters to create a link.
  7. Save the popup and test it out.

 

And here are the same steps in a short animation.

 

Editing Web Forms and Popups

 

Embedding an edit web form within Operations Dashboard for ArcGIS

 

The idea with this one is to add an embedded content element within the dashboard, and configure it to display your web form.  The URL of the web form will include the mode parameter set to edit and the objectId will be dynamically populated.  Using a filter action triggered by other element within the dashboard, such as a list, the embedded content element will refresh your web form with the relevant record to be edited.

 

At a high level, the steps are as follows:

 

  1. Set a refresh rate on a web map displaying your survey feature layer.
  2. Add your web map to a new dashboard and configure a list element to display survey records.
  3. Add an embedded content element. In the Data options, set the Type to Features and add the URL of your web form with the mode=edit  and objectId parameters. Make sure the objectId parameter is set to pull a value from the ObjectID column in the fields collection.
  4. Next configure a filter action on the list element and target your embedded content element.
  5. Click on a record in the list, and your web form in the embedded content element will display your record in edit mode.

 

The animation will guide you through the whole process. Click on the animation to make it larger.

 

Editing Web Forms in Dashboard

 

Including an edit web form link in an e-mail.

 

In this scenario, we will use a web hook to automatically send an e-mail to the user submitting the survey. This email will contain a link for the user to complete or make modifications at any time.  To do this you must be familiar with web hooks in general or with either Microsoft Flow or Integromat.   Like in the examples above, the trick is to be able to dynamically populate the objectId of the record you want to edit. Luckily enough, the payload of a Survey123 web hook includes information about the user submitting the data (if logged-in) as well as responses to questions in the survey. So as long as the user is logged-in or a specific question in the survey captures the contact e-mail, you can make this work.

 

Here are the high level steps:

 

  1. Create a new survey. If you plan to share your survey publicly, include an e-mail question and make it required. If  you plan to keep your survey secured, you will be able to get the e-mail from the logged in user's profile.
  2. Use Microsoft Flow or Integromat to automate e-mail notifications when your survey is submitted.  The objectId of the record submitted is included in the payload of the web hook. Simply add the objectId to the URL of your edit web form when you add the link in the body of your e-mail.

 

Here is some visual aid in case you are using Integromat.

 

Integromat

 

If you are using Integromat or Zapier, the pattern is pretty much the same: use the webhook payload to retrieve the objectId of the submitted record and use if for your edit web form url.

In this blog post, I describe some common techniques that will help you better understand and hopefully improve the quality of location data you get from field users.  Location data quality depends on multiple factors, including field conditions, field user skills and the hardware used. Having said this, a good smart form design can do a lot to help get the best possible quality location data.

 

This post focuses particularly on field data collection workflows where you will rely on integrated or external GNSS receivers to capture location data. It assumes familiarity with XLSForms, Survey123 Connect and the Survey123 field app.

 

Persisting Location Accuracy (Part 1)

 

If location quality matters to you, you should consider systematically storing the location accuracy of every record captured.  This is just as true for formal high accuracy field data collection workflows by professionals, as it is for casual field data collection by volunteers.  Horizontal accuracy is about the simplest and most important quality indicator of your location data.  Having field users understand the horizontal accuracy while they collect data is not enough; you need to persist it for every record captured, so you can analyze it later.

 

In XLSForm, you can store the horizontal accuracy as reported by your field device as follows:

 

typenamelabelcalculationbind::esri:fieldType
geopointlocationGPS Location
calculateaccuracyHApulldata("@geopoint",${location},"horizontalAccuracy")esriFieldTypeDouble

 

The key is the calculation in the second question of the survey.  It automatically gets the horizontal accuracy from the geopoint question and stores it in a field named accuracy. Since you do not want end users to overwrite this value, the sample above keeps the output of the calculation in a question of type calculate. Calculate questions are not presented to the end-user in the form, but their values can be be stored in ArcGIS.  In this particular case, since the horizontal accuracy is expressed in meters, the bind::esri:fieldType column has been set to esriFieldTypeDouble.  If the esri:fieldType value is not set explicitly, Survey123 will automatically store the calculated value as text.

Horizontal accuracy values are always expressed in meters.

        

I like to add a calculate like the above to every survey I prepare for field data collection. Looking at a dataset from field users without location accuracy information is not very comforting.  Without location accuracy information in a GIS dataset, what level of confidence can you have on those points in the map?  Capturing the horizontal accuracy like described here adds absolutely no overhead, and it always pays off.

 

In the next screenshot, locations captured along a trail have been buffered to visually represent the reported horizontal accuracy.  This simple visualization can provide you with great information about the quality of locations in your dataset. Before performing GIS analysis, you can set criteria to filter out records that do not meet a location accuracy threshold of your choice.

 

 

You can also use the location accuracy information to better understand in what conditions poor location data is created. You can analyze spatial patterns, identify particular users or devices that do not meet minimum standards.

 

Persisting Location Accuracy (Part 2)

 

The horizontal accuracy that Survey123 returns from a geopoint value is provided by the location sensor you are using.  On top of the actual value (say 6.2 meters for example), it is important to understand the level of confidence level for that value. That is, with what level of confidence will the location be within a radius of the horizontal accuracy reported (say within a radius of 6.2 meters of the point for example)? 

 

The Google Developer documentation, defines horizontal accuracy as the radius of 68% confidence. This is like saying that for Android devices, there is a 68% chance that the true location is within a radius equal to the reported horizontal accuracy. Apple has not documented their definition of horizontal accuracy.

 

When using external high accuracy GNSS receivers with Survey123, the confidence interval can be more accurately be defined if the receiver is able to report what is known as root mean square (RMS) accuracy. For RMS accuracy, the default CL (Confidence Level) is 68%. Some organizations require reporting accuracy with a 95% CL. If this is the case, the United States Federal Geographic Data Committee (FGDC), through its National Standard for Spatial Data Accuracy, establishes that the following factors should be applied:

 

  • 1.7308 for horizontal accuracy.
  • 1.9600 for vertical accuracy.

 

The conversion between RMS 68% CL and 95% CL applying the factors above should only be calculated, again, if the GNSS receiver is reporting horizontal accuracy using RMS.  In the Direct GNSS external receiver support in Survey123 3.3 blog post you can find an example of how you can use RMS horizontal accuracy values into a 95% confidence level.

 

For both built-in and external GNSS receivers, you can complement the reported horizontal accuracy with other commonly available location metadata such as the speed and also the position source type (Integrated GPS, external GPS, user-defined, etc). For a quick reference of other location metadata fields available, have a look at the Pulling data from geopoint questions blog post.

 

If you use external location sensors, such as high accuracy GNSS receivers, then you will be able to get a lot more location metadata with Survey123 such as the fix type (GPS, differential GPS, RTK fixed, RTK float, SBAS...), differential age, number of satellites used, etc. The complete reference guide of all location metadata you can extract is in our Geopoints—Survey123 for ArcGIS | ArcGIS help topic.

 

Persisting the location accuracy as a GIS attribute is generally a good idea, but it does not help field users understand how to to get a better location with Survey123.  Next, I will discuss a handful of techniques that will help end users actively contribute to higher location quality data.

 

 

Location Accuracy Threshold

 

The location accuracy threshold is an XLSLForm value expressed in meters.  When the accuracy threshold is not met, the map panel in the survey will be automatically highlighted in red and the location icon will spin.  This will give the field user a visual clue indicating that (1) a better location is desired and (2) that the Survey123 app is actively trying to get new locations to meet the threshold. 

 

Setting the location accuracy threshold in your smart form is a very simple way to visually alert end users when the location is not considered accurate enough.  A red map panel is an indication to end users that it is best to stay put and wait until the device is able to get a better location.  If for whatever reason the device cannot get the desired accuracy, the end user can manually tap on the spinning location icon to fix the geopoint.

 

The accuracy threshold is set, in meters, in the body::accuracyThreshold XLSForm column as shown below.  This column accepts integers and decimals, but not XLSForm expressions. That is, the accuracy threshold is a fixed value.

 

typenamelabelbody::accuracyThreshold
geopointlocationGPS Location5

 

It is important to train end users so they understand exactly what is happening when the map panel is red and the location icon spins.  While the location icon is spinning, Survey123 is actively retrieving new locations from the device. If the end user moves, the location will change!  This is very different to the typical behavior of the Survey123 field app, which by default will immediately fix the location after the form is open.

 

A conservative value on your accuracy threshold, can help reduce situations where field users report location data that is clearly below the capabilities of their device.  For example, when a device has been off for a while, the GPS may need some extra time to warm up before getting the best results.  If you know that in normal conditions, you can get your devices to report horizontal accuracy in a range between 5 and 10 meters, then you can conservatively set the accuracy threshold to 20 meters. This will cause the map panel to turn red only in exceptional cases, letting the user know that it is best to stay put and wait a bit in order to let the device get a better fix.  In normal conditions, the accuracy threshold will be met after a few seconds, the map panel will turn green and the location will be fixed. Figuring out such conservative value for the accuracy threshold requires some trial and error. It will vary depending on the range of devices used and average field conditions. 

 

You can also go with a more aggressive approach where the accuracy threshold is set closer to the limits of what field conditions and your hardware can typically provide. In this case, you will be using the accuracy threshold to constantly remind end users of the importance of getting the best location possible. With an aggressive accuracy threshold value, some users may not be able to get a horizontal accuracy value below the threshold.

 

Always be aware that the location accuracy threshold value by itself will not prevent users from submitting data below the threshold. If you do not want data to be submitted unless a specific accuracy threshold is met, then you will need to use constraints. We will describe this later in this blog.

 

Location Quality Expressions

 

Using XLSForm syntax you can build sophisticated validation rules for location data.  These XLSForm expressions can take into account the horizontal accuracy, and many other properties of your location data. You can also use other form data in your expressions. Here are some examples of location quality expressions you can build:

 

XLSForm expressionDescription
${speed}<0.2Speed is less than 0.2 meters per second.
${positionSourceType}=3Location has been captured with an external location sensor
${fixType}=4 and ${positionAccuracy}<1Location is RTK fixed and mean radial spherical error is less than 1 meter.
${differentialAge}<3Location's differential age is less than 2 seconds

 

You can use XLSForm expressions like the above in different contexts. For example, you can use them to show visual warnings to the field user using notes. See the Understanding Notes in Survey123 for details on using notes for this purpose.

 

You can also use apply these expressions to the constraint and bind::esri:warning columns of a geopoint question. If you apply them to the constraint column, the end user will not be able to submit the survey unless the expression evaluates to true and the map panel will be highlighted in red.   This is what is known as a hard constraint.  If you apply them to the bind::esri:warning column, the map panel will be highlighted in yellow when the expression is not met.  This will simply give end users a visual clue indicating that something is wrong with the location data.  That is a soft constraint.

 

 

The contraint_message and bind::esri:warning_message columns are used to provide users with a specific error or warning message. Here is a complete example illustrating how to use the bind::esri:warning column to display a warning if the location has been captured while moving at a speed larger than 0.2 meters per second.

 

typenamelabelcalculationbind::esri:warningbind::esri:warning_message
geopointlocationLocation${speed}<0.2Please do not move while fixing the location
calculatespeedSpeedpulldata("@geopoint",${location},"speed")

 

Typically, a location quality expression is going to validate some properties of your location data. For example, its speed, accuracy, etc. These properties are obtained through the pulldata() function. It is not good practice to combine the pulldata() function with any other XLSForm operator or function.  In the example above, I created a calculate question to get the user speed from the geopoint object first, and then I referenced that value in the bind::esri:warning column. In this way, the pulldata() function is left alone in its own question avoiding any potential issues.

 

Expressions in bind::esri:warning define soft constraints. Even if the condition is not met, users will be allowed to submit the survey.

 

In the next example, I use the constraint column to apply a hard constraint. The user will not be able to submit the survey unless the accuracy is less than 1 meter and the fix type is RTK.

 

typenamelabelcalculationconstraintconstraint_message
geopointlocationLocation${fixtype}=4 and ${pa}<1RTK fix and sub-meter accuracy are required.
calculatefixtypeFixpulldata("@geopoint",${location},"fixType")
calculatepaPApulldata("@geopoint",${location},"positionAccuracy")

 

In this example, I use a constraint to force users to use a GPS to set the location of the geopoint. If the user manually changes the location in the map, Survey123 will not let the submission go through.

 

typenamelabelcalculationconstraintconstraint_message
geopointlocationLocation${source}!=1
You can't manually set the location. Use the GPS.
calculatesourceSourcepulldata("@geopoint",${location},"positionSourceType")

 

Expressions in the constraint and warning columns are evaluated every time the value of the question changes. If you add expressions to these columns in a geopoint question, they will get evaluated as soon as the location is updated. 

 

Summary

 

A good smart form design can help field users capture better location data. Using XLSForm expressions we can actively check the quality of location data in the survey and provide users with clues to help correct issues.  XLSForm expressions can also be used to prevent users from submitting location data that does not comply with our own business rules.  Finally, XLSForm expressions allow us to store many location quality indicators as GIS attributes.  This information is very valuable to support QA/QC workflows and before analysis and visualizations of the data are performed.

New Britain is the largest island in the Bismark archipelago of Papua New Guinea. William Dampier, takes credit for being the first British man to land on this island. This happened on February 27, 1700. Since then we call this island New Britain (Nova Britannia at the time). Both Wiliam Dampier and New Britain are fascinating. This island, of volcanic origin, was populated by two main indigenous groups: the Papuans who arrived tens of thousands of years ago, and the Austronesians, who joined the party around 2000 years ago. The island hosts remarkable traditional cultures. Still today, over 10 different Papuan languages  are spoken in the island and as for the Austronesians languages, we count more than 40! William Dampier is known for being the first man to circumnavigate the world three times. He grew as a pirate attacking Spanish ships in America and then became a successful navigator and explorer. Dampier documented winds, tides and currents that were critical for future explorations by James Cook and Horation Nelson. His natural history observations were also highly influential on works of Alexander Von Humboldt and Charles Darwin.

 

With our 3.3 update, we celebrate the adventures of William Dampier and his landing over 300 years ago to the island today known as New Britain.

 

 

Enhanced support for GNSS receivers in the Survey123 field app

 

Starting with version 3.3, you can establish a direct connection between the Survey123 field app and your GNSS receiver. This is useful for several reasons:

 

  • More accurate location data:  By leveraging dedicated high accuracy GNSS hardware, you can substantially improve the quality of the location data you capture with your smart forms.  There is a wide range of GNSS receivers, some of them able to produce, if used properly, sub-meter and sub-centimeter accuracy. 
  • Better understanding of the quality of the data: Having more accurate location data is not enough. It is also important to understand how accurate that data is.  Most GNSS receivers are able to provide important information that describe how the locations you are getting were produced. This includes things like the type of fix (RTK Fixed, RTK Float, GPS, Differential GPS, SBAS...), the differential age, reference station ID and many others.  With version 3.3 of Survey123 and above, you can select which of these properties are important to you and store them as GIS attributes in the feature layer columns of your choice. For a complete list of all the location metadata you can extract from your location data with Survey123 refer to the extract geopoint values help topic.
  • Custom validation rules: You can use all the GNSS metadata described above to create custom data validation rules that get triggered while the data collection is happening. This is critical, as it can save many trips back to the field! It is important to note that you can use all the power of XLSForms to build sophisticated expressions that take advantage of this metadata.

 

In this initial release, we have thoroughly tested GNSS receivers built by a number of vendors including Leica, Eos Positioning Systems, Bad Elf, SXblue, and Trimble, although it technically can work with other receivers as long as they output NMEA sentences. Our choose a receiver help topic includes more information regarding currently supported hardware.

 

To support direct connections to external GNSS receivers, you will notice we have introduced several changes in the Survey123 field app. The Settings dialog in the app now includes options to connect to external location sensors via Bluetooth and network ports. For each connection you can also set user preferences such as a custom name for your device, the height of the antenna and handy visual and audio alerts to warn users in case the connection between the GNSS receiver and your device is interrupted.

 

Our XLSForm support has also been expanded so you can include in your form expressions to help work with the GNSS metadata in real time.  This really opens a wealth of possibilities for you to use this information to design soft and hard data validation constraints in your forms.  The geopoint help topic in our doc describes in more detail how to take advantage of this.

 

Point to Point Measurements with Spike® and the Survey123 field app

 

We introduced support for the Spike laser range-finder in March 2018.  Spike is an excellent complement to Survey123, allowing you to quickly bring laser-accurate measurements into your forms. Jim Moore and Brett Stokes, who recently joined the Survey123 team, put together this video to demonstrate the basics of bringing measurements from Spike into Survey123.

 

 

In this release, we are bringing support for the Point to Point (P2P) measurement mode which was missing in our initial implementation. P2P is a very handy way to calculate distances between two objects because it does not require you to identify a 2D plane on which to measure. In P2P mode you can simply aim Spike at the first object, such as the side of a house, and take a photo. Then, aim at the second object, such as a tree or fence, and take a second photo. Spike will calculate the distance between those two objects and bring that value into your Survey123 form. You can also use Point to Point to measure the height of an object such as a tree, streetlight or building.

 

If  you are already using Spike with Survey123 and wish to try Point to  Point measurements, make sure you upgrade your Spike app from the store  with the latest version.

 

For more details on how Spike and Survey123 work together, check out theUsing Spike® with Survey123 for ArcGIS blog post.

 

Date/Time question type in web designer

 

A new Date/Time question type has been added in web designer.  This will help you capture both a date and a time in one single row, saving vertical space in your forms. Within web designer you can set a default value for this question, either selecting a static value, or the exact time when the form is open.  You can also set validation rules to constrain inputs to a time window.

 

Date-Time Question type in Survey123 web designer

 

Better field and choice naming in web designer

 

If you never liked auto-generated field names such as field_0, field_1... in your Survey123 feature layers, you will want to pay attention to this one. Let me start with a bit of context first:  When you publish a survey in web designer, we create on your behalf a feature layer field for every question in your form.  These fields are used later to store responses to your survey. Up until this release, the name of these fields followed a simple field_0, field_1 pattern. This was typically never an issue when using the Survey123 web site to inspect your data, but it was not ideal when exporting results into Excel, CSV, or when configuring web hooks.

 

Starting with this update, a more friendly name will be given to these fields based on the label of your questions. The following animation compares what an auto-generated feature layer schema looked like before, compared to what you will get now.

 

 

Field and choice names are now derived from your question and choice labels. Given that there are some constraints that apply to the names that can be given to fields in a feature layer, you will notice that your labels will not be used literally. For example, spaces will be replaced with underscores (_), very long labels will be shortened and special characters will be eliminated.

 

Using the Modify Schema dialog you can optionally fine tune the names of fields and choices to your liking, although remember that you will not be able to make adjustments in the schema once the fields have been added to the feature layer.

 

The option to Modify Schema in the Publish Survey Dialog has also been made more visible as a reminder of this important step before you publish your survey.

 

 

Delete and edit existing records for survey owners

 

As a survey owner, I am sure you always submit a couple of records to test out your survey designs, immediately checking values in the Data tab. You can now easily delete, and even edit values right from the Survey123 web site. You no longer need to use a separate tool to do that. All you need to do is to select the records in the table view, and then choose the Delete or Edit options in the right panel to make the changes you want.

 

Edit and delete records, right from within the Survey123 web site

 

At  this moment in time, the option to edit records will not be available  for surveys published from Survey123 Connect, but we are looking to  relaxing this limitation in the future.

 

 

 

Other fixes and enhancements

 

Survey123 web site:

 

  • ENH-000107831 Provide the ability to edit the survey submission "Thank you" screen seen in the browser when the survey is published from Survey123 Connect
  • Import custom themes from other surveys.
  • For text questions, in the Analyze tab, you can now choose to display a list of word counts or complete responses.
  • Much improved user experience for configuring custom report templates.
  • BUG-000119133 Field aliases in a survey in the Survey123 for ArcGIS application are not respected in single choice and drop-down lists.
  • BUG-000098360 When downloading data from a Survery123 survey that was made using Survey123 web designer, the field names are converted to {field_1}, {field_2}, ... instead of honoring the name in the data table.
  • BUG-000103150 Surveys created in Survey123 web designer using drop-down choices display 'choice1, choice2, ...' when exported to a CSV file or a shapefile.
  • BUG-000105778 Survey question choices from single choice questions do not correctly translate when exported to CSV or shapefile from surveys that were created via Survey123 web designer.
  • BUG-000114116 When using a custom template that writes out results from related and non-related attributes from a survey, the non-related records do not populate the template table.
  • BUG-000113548 When an audio attachment is included in the survey, the Survey123 for ArcGIS website fails to generate a report.

 

Survey123 field app and Connect

 

  • Allow users to zoom beyond level 18 if the tile package includes more detailed levels.
  • BUG-000119481 When creating a Survey123 for ArcGIS survey with an existing feature service, the float or single field types do not auto populate in the XLSForm created
  • BUG-000110786 When signing in to the Survey123 for ArcGIS (iOS) app configured with an IWA portal, the app uses the username and password entered at the beginning of the app session for all subsequent sign-in attempts.
  • BUG-000119847 Date questions in certain languages are not displaying the calendar correctly
  • BUG-000119899 Integer type questions with the "spinner" appearance type do not work
  • ENH-000105170 In Survey123, allow users to change the Location Provider, from the internal receiver to an external receiver.
  • ENH-000109937 Provide an indicator within the Survey123 interface to display which GPS unit the app is using.

In this blog post I will describe how you can integrate Survey123 with Spike. Spike is a laser-based measurement solution by Esri partner ikeGPS.   What makes Spike unique is that it allows you to easily measure the dimensions of remote objects, such as areas and lengths, all from a photo.

 

Say, for example, while doing a property inspection you need to calculate the height of a balcony, or the size of a garage door.  Rather than using a ladder and measuring tape, you can simply take a photo of the building, and then calculate dimensions by drawing on top of the photo.  You can similarly use Spike to calculate the size of road signs, billboards, poles, etc.   

 

You can embed Spike questions within Survey123 and automatically bring all measurements from Spike into your form: Tap on a Spike question, snap a photo, measure on it… and your measurements will get persisted in ArcGIS along with the rest of the survey data.

 

The integration of Survey123 with Spike is exciting news, because it has the potential to boost the productivity of certain data capture workflows tremendously.  This short video tutorial describes the basics of how Survey123 and Spike work together:

 

 

Lets next get into the details...

 

 

About Spike®

 

The Spike laser: The most visible aspect of Spike is the laser instrument. It is an eye safe Class-1 laser, designed to measure distances in a range between 2 and 200 meters with an accuracy of 5 centimeters. The laser is designed to be attached to your smartphone or tablet and paired using a Bluetooth connection.

 

Spike units have a retail price of $499 and are sold online and through a network of resellers.

 

The Spike app: Once the laser is paired, the Spike app uses the camera on your smartphone or tablet so you can take a photo of your object of interest. Using the distance to the photographed object captured by the laser, the Spike app can accurately calculate lengths and areas as you draw lines and polygons over the photo.

 

The Spike app is available for iOS and Android operating systems and is certified for use against a wide collection of smartphones and tablets

 

Getting familiar with Spike: Spike is straight-forward to use, but knowing a handful of tips and good practice in the field is key to get the best out of it.  The best reference to take your first steps is the Spike User Manual

 

Spike can be operated through two main modes:

 

  • Photo Measurement: When working in New Photo Measurement mode, you take a single photo, and then derive all measurements from it.  You can for example simply get the distance from your location to the object, or extract lengths and areas by drawing on your photo.
  • Point-to-Point Measurement: In this mode you can calculate the distance (horizontally and vertically) difference between two objects by taking a separate photo of each object.

       

Photo Measurement mode:In its simplest form, the Photo Measurement mode will allow you to simply snap a photo of an object, and get back the exact distance to that object form your current location.  This alone can be useful in multitude of use cases.  Say for example you need to document the width of a road: stand on one side of the road and take a photo targeting at the other end across. After taking the photo, Spike will give  you the exact distance. Not only you can get an accurate measurement, but you can do it without risking your life; safely from one side of the road. 

 

With Photo Measurement mode you can also measure distances (and areas) on objects by simply drawing on top of the photo you just took. For example, say you want to know what is the height of a building. Well, you take a photo of it, and then you extract the height by drawing on that photo.  If you want to derive measurements from a photo in this way, certain conditions need to be met.  When used correctly and in ideal conditions, you can expect a measurement with an accuracy of 1%.  For example, if you measure the height of an object to be 10 meters, you can expect the accuracy to be around 10 centimeters.

 

Let’s describe next what used correctly and ideal conditions means:

 

  • Measurements against a flat surface: You should only perform measurements on flat surfaces, such as a wall or a sign. The specific shape you want to measure should be on a rectangular plane. This is because you will need to define the plane in the photo before you can measure anything. 
  • Face your target front-on: For the most accurate measurements, you should take your photo while facing your target perpendicularly. That is, you need to be right in front of the plane on which you will do your measurements. If you need to take your photo at an angle, the expected accuracy of the measurements will be reduced progressively as you get further from being perpendicular.

 

 

  • Ideal target range: The ideal working range of the laser is between 6 and 100 meters, although technically you can push the limits and measure objects between 2 and 200 meters away.
  • Center your target: You also want to make sure that the object you want to measure is right in the center of the camera bulls eye, because the laser always calculates distances through the bulls eye.
  • Draw accurately: Once you have taken the photo, you will want to make sure you accurately draw your lines and areas for measurement. The Spike app has a handy magnifying window that will help you with that task.   If your photo is taken following all the tips above, but you do not take pride in carefully drawing the exact shapes you want to measure… the measurement will not be as accurate as it could be.

 

So far all of this may sound a little bit intimidating, but once you have the Spike device in your hand things are much simpler. Not just simpler, but also fun. Next you can see Brandon having a good time while ‘spiking’ outside Esri.

 

 

From our own experience, the Photo Measurement mode works best with man-made structures: walls, signs, billboards, doors, straight poles, streetlights etc.  Measuring the height of trees, rocks etc is much more challenging because you often do not have a good plane to reference your measurements.  If you do not have a good 2D plane to make your measurements on, then you should consider using the Point to Point measurement mode.

 

Point to Point Measurement mode: Point to Point or P2P is very useful when all you need is to measure the distance between two objects in front of you and you either cannot fit the two objects within a single photo, or when you do not have a good 2D plane on which to base your measurements. With P2P, you simply need to center your camera on your first target, say for example a tree and take a photo. Then you turn from that same location to take a photo of your second target, say for example a wall. Spike will compute the horizontal and vertical distances between the two photographed objects.

 

Like with any tool out there, it takes a bit of practice to fully understand what is the best measurement mode for the job.  Spike can greatly boost the productivity of field workers saving valuable time. It also can provide a much safer way to make measurements. For hard to reach objects, Spike can make it possible to perform measurements that could not be done otherwise.

 

Now that I have described the basics of Spike, it is time to go into the details of how Survey123 and Spike work together.

 

Integrating Survey123 with Spike (Photo Measurements)

 

Starting with Survey123 version 2.6, you can incorporate Spike functionality into your form. This allows you to streamline the use of Spike within your data collection workflows. For example,

if you have a survey to help map the presence of graffiti in the city, you will want to capture many things including the location, typology (Tag, Throw-up, Blockbuster…), surface, photo and size.  Spike can help you capture the size accurately, so you can estimate the cost of removal.

 

Adding Spike questions into a survey: You can add Spike questions to a survey using Survey123 Connect for ArcGIS. In your XLSForm, add a question of type image, and then set the appearance to either spike-full-measure, spike or spike-point-to-point:

 

  • spike: With this appearance, Survey123 will launch the Spike app so you can simply aim at your object of interest and snap a photo of it.  The spike appearance is useful when you simply want to get the distance to the photographed object and its estimated location.
  • spike-full-measure: This appearance sets the Spike app in New Photo Measurement mode as described in the previous section. When using this appearance, you will be able to get the measured lengths and areas, into your form.
  • spike-point-to-point: Launches Spike in Point to Point mode so you can calculate the horizontal and vertical distances between two photographed objects.

 

When using the spike-full-measure, spike-point-to-point and spike appearances, Survey123 will let users easily launch the Spike app from within the form.  Once the photo has been taken and measurements added, Spike will return back to Survey123, so all measurements of interest can be extracted and stored in the survey.  The measure object with Spike help topic describes step by step how Survey123 and Spike work together.

 

Spike photos will be stored as an attachment automatically.  This is a great feature because photos will become a verifiable record persisted with your data. Photos will show how accurately the measurement was made in the field, and if needed, the photo itself could be used in the back-office to make additional measurements.

The use of Spike within Survey123 is only available to logged-in users. Spike questions can be used while online or offline, but you must be logged-in.

                            

 

Extracting Spike measurements with pulldata(): Spike uses EXIF photo headers to store information such as the location from which the photo was taken, the direction, distance and measurements made in the Spike app.  In a previous blog post we explored how to extract EXIF metadata from an image in Survey123 using the pulldata() function. This same technique can be used to extract Spike metadata.

 

The Spike metadata in the EXIF of the image is split between standard EXIF tags and custom information stored in the ImageDescription tag. The following table describes standard tags populated by Spike:

 

 

EXIF tagDescription
gpsdestlongitude

The estimated X coordinate of the photographed object

gpsdestlatitude

The estimated Y coordinate of the photographed object

gpsdestcoordinate

The XY coordinate pair of the photographed object

gpstimestamp

The time when the photo was taken

gpsdatestamp

The date when the photo was taken

gpsdestdistance

The distance between the photographed object and your device

imageDescription

A JSON string including measurements captured by the Spike app

 

This expression will extract the distance in meters between the photographed object and the device:

 

pulldata("@exif", ${spike}, "gpsdestdistance")

 

In order to extract photo-measurements from Spike, your pulldata() function will need to first get the ImageDescription EXIF tag, and then parse the returning JSON string to extract the specific dimensions you are looking for.

 

For example, the following expression assumes the first measurement made in Spike was of type length, and gets its value in meters:

 

pulldata("@json",  pulldata("@exif", ${spike }, "ImageDescription"), "Spike.objects[1].dimensions")

 

The next expression assumes the first measurement made was of type area and extracts its value in square meters:

pulldata("@json",  pulldata("@exif", ${spike_full}, "ImageDescription"), "Spike.objects[1].area")

 

This expression gets the horizontal distance between two objects as measured through the Point  to Point measurement mode:

 

pulldata("@json",  pulldata("@exif", ${spike }, "ImageDescription"), "P2P.horizontal_distance")

 

 

Spike returns many other properties through the ImageDescription EXIF tag including: the pitch, roll and direction of your device when the photo was taken, the unique identifier of the specific Spike laser unit that captured the distance measurement etc. The Spike section of the Media help topic describes in detail the contents of the JSON object stored in the ImageDescription.

 

Another way to explore the contents of the ImageDescription tag is to simply add a text question to your form and have it populated with the contents of the ImageDescription tag through a calculation.  When you do this, it is important that you set the esriFieldLength column of your text question to 1000 characters. The ImageDescription JSON string can be quite big.

 

A great starting point to experiment with all the things you can do with XLSForms and Spike is our Spike sample. I strongly recommend that you have a look at it:

 

1.    Open Survey123 Connect for ArcGIS
2.    Click on Create New Survey
3.    Search for the Spike survey in the Samples category
4.    Publish it and test from your smartphone
5.    Look into the XLS file to learn all the techniques used in the sample.

 

Integrating Survey123 with Spike (Estimate Locations)

 

While the heart of what Spike excels at is helping you measuring dimensions over photos, you can also use Survey123 to roughly estimate the location of remote objects.  For example, you can map the location of a building without stepping out of your car, or the location of a streetlight across the street.

 

Before we go into the details of estimating the location of remote objects, I want to emphasize that Spike is not a high accuracy surveying device.  It can measure accurately distances to objects, but when estimating the location of remote objects there are several sources of error that will come into play outside of what Spike is designed to do.

 

To define the location of a remote object, Spike relies on the following:

 

  • The location of your own phone or tablet: This is the first source of error. A typical smartphone is not going to give you a highly accurate location for your phone. Your location accuracy may vary depending on many factors and if your location is off, the estimated location of the remote object will be off too.  To correct this problem, you can connect an external GNSS receiver to your own phone or tablet via Bluetooth. If used correctly, external GNSS receivers will improve the source location used by Spike in its calculations. In the side photo, for example, Survey123 is combined with Spike and Juniper Systems Geode Sub-meter GNSS antenna.
  • The distance to the remote object: This is going to be very accurate as long as you keep yourself within the working range of Spike (2 to 200 meters) and you aim at the remote object using the bull’s eye in the Spike app. Measuring distances to objects is what Spike is designed to do, and does it very well.
  • The azimuth to the remote object as measured from your own device location: The azimuth is another important source of error. The azimuth is obtained from the compass in your own smartphone or tablet. The compass in your smartphone must be calibrated before use by Spike. It is critical that you follow steps in this compass calibration video-tutorial before you estimate remote locations. Even with your compass calibrated, it is normal to get deviations of a few degrees, which will increase the error as the distance to the remote object increases.

 

The gpsdestlongitude and gpsdestlatitude properties will be populated with both the spike and spike-full-measure appearances.If you do not plan to capture any length or area measurements and all you want is the location of the photographed object, then the best is to use the spike appearance.

 

To extract the estimated latitude and longitude of the photographed object, you will use the pulldata() function within a calculation. For example, the following expression will extract the latitude:

 

pulldata("@exif", ${spike}, "gpsdestlatitude")

 

You can store the latitude and longitude as attributes in your survey, and then create an XY Event Layer with them in ArcGIS Pro or other tools.  

While it should be possible for you to calculate a geopoint question with the latitude and longitude values extracted from the Spike photo, there is currently a known limitation in Survey123 preventing the successful calculation when applied. This is in our backlog and we will work towards resolving the problem as soon as possible.

          

As stated before, it is best that you exercise Spike in the field in all its capabilities. To fully understand its potential and limitations you must experience it in the field in real working conditions.

 

If you need some ideas, here is an informative video by IkeGPS further describing some common use cases for Survey123 and Spike.

 

Our Developer Summit conference is approaching fast.  We at Esri always put a lot of energy into this event. It is an opportunity to meet face to face with many of you, share our work, discuss new features and road map plans. 

 

2019 will be very special for us as we will make 4 years since we announced the graduation of Survey123 from an Esri Labs proof of concept into a full fleshed ArcGIS Product. This year we will announce details on ambitious plans for the new releases to come: direct connections to external GNSS receivers in the field app, added support for polygon and line features, a release date for our Survey123 report services, enhanced integration with Spike laser measurement tools, a new extensibility framework and many more great news. Members of the Survey123 team from our offices in Beijing, Australia and Redlands will come to Palm Springs. We are really excited for the opportunity to meet you again.

 

Today we want to share with you some tips and a guide for anything Survey123 at these events.

 

The Events App

 

I highly recommend that you download the Esri Events app, so you can always be on top of last minute changes and manage your schedule.  As of the time when I write this post the detailed schedule is not yet up in the app, but it will be available shortly.  This app will be a good companion throughout the entire conference.

 

Pre-Summit Hands-On Training

 

Our Esri Pre-Summit Hands-On training sessions take place on Sunday 3 and Monday 4. These are full day sessions with a mix of lessons and practical challenges. You bring your own laptop, so you can take all homework home. Massive learning guaranteed!   Our sessions start at 8:30 sharp. You can register starting at 7:30 at the Hotel Riviera.

 

  • Mastering Smart Form Design with Survey123 and XLSForm.
    • Sunday, March 3 from 8:30am to 5:00pm
    • Royal II at Hotel Riviera Palm Springs.
  • Extending Survey123 for ArcGIS.
    • Monday, March 4 at Royal II from 8:30am to 5:00pm
    • Royal II at Hotel Riviera Palm Springs

 

Esri Showcase and GIS Solutions Expo

 

The GIS Solutions Expo area opens Monday from 2:00pm to 7:00pm at the Primrose Ballroom. This is a great place to meet many partners who work closely with Survey123.  Here are a few:

 

  • Microsoft. Diamond sponsor this year. Showcasing the Survey123 Microsoft Flow connector.
  • Bad Elf, EOS Positioning Systems, Trimble, GENEC:  Providing GNSS receivers for which Survey123 3.3 will provide direct connectivity capabilities.
  • Zebra Technologies: Manufacturing rugged devices compatible with Survey123.
  • Geo Jobe: Custom software development services.

 

During this time, you can also swing by the Esri Showcase area to visit the Survey123 team in the Apps area.

 

On Tuesday from noon to 8:30pm, Wednesday from 1:00pm to 6:30pm and Thursday from 9:00am to 4:00pm the Esri Showcase will continue open in Oasis 1 and 2. Come to the Showcase and look for the Survey123 booth. This is your chance to meet people from the Survey123 team, have a relaxed conversation, ask questions and network with other Survey123 users.

 

Meet the Teams

 

The Meet the Teams event will take place from 6:00pm to 8:30pm in Oasis 1 and 2 on Tuesday. In other words, at the Esri Showcase area. Drinks, food and GIS to spare.

 

Esri Dev Summit Technical Workshops

 

These are 60 minute sessions. All take place in the Palm Springs Convention Center. A good place to learn from the best, ask questions and meet other Survey123 users.

 

Other related Technical Workshops

 

If you plan to extend or customize Survey123, the AppStudio sessions are a must.

 

 

Survey123 Demo Theaters

 

Short and to the point sessions focusing on very specific topics. Live demos and time for questions are guaranteed. All Demo Theaters take place at the showcase area.

 

 

Survey123 Special Interest  Group (SIG)

 

This session is a great place to mingle with other Survey123  Ninjas and members of the Survey123 team. Keep your own lunch box because this SIG takes place at noon. We will break the ice with help from a special guest who will share his own work deploying Survey123. Then we will open an interactive Q&A session.

 

  • Survey123 SIG
    • Thursday, March 7 from noon to 1:00pm
    • Primrose A at the Palm Springs Convention Center