ArcGIS Survey123 Blog

cancel
Showing results for 
Search instead for 
Did you mean: 

Other Boards in This Place

Latest Activity

(193 Posts)
IsmaelChivite
Esri Frequent Contributor

We are happy to announce our first update to ArcGIS Survey123 in 2021. This new release includes fixes and new features, including a new gallery of survey templates in the web designer, much faster lists in the field app and greatly improved support for custom JavaScript functions in Survey123 Connect. Read more for the details!

 

Read more...

more
6 10 512
IsmaelChivite
Esri Frequent Contributor

This article provides a quick reference to help you work with Survey123 attachments in Microsoft Power Automate. It describes how you can include Survey123 attachments in an email and how to upload attachments into Microsoft OneDrive.

Read more...

more
2 4 711
IsmaelChivite
Esri Frequent Contributor

This article describes some advanced techniques to help you work with multiple Survey123 attachments in Integromat.

Read more...

more
1 2 264
IsmaelChivite
Esri Frequent Contributor

The Survey123 website and web app have just been updated. Learn what's new in this article.

Read more...

more
4 0 569
IsmaelChivite
Esri Frequent Contributor

With each Survey123 release, we assess the platforms we support and adjust these based on customer needs and technology trends. The purpose of this article is to provide as much advance notice as possible regarding these changes.

Read more...

more
3 1 587
GlenShepherd
Esri Contributor

Setting boundaries for survey submission using JavaScript in Survey123

Read more...

more
8 3 575
IsmaelChivite
Esri Frequent Contributor

New ranking question, enhanced choice filters, hidden appearance, improved report template syntax and much, much more in this release.

Read more...

more
9 11 4,666
IsmaelChivite
Esri Frequent Contributor

The best way to tackle a big problem is to break it down into smaller ones. Lists are no different. Choice filters let you leverage user-entered values in your form to dynamically control which elements of a list are shown or hidden. The most common use for a choice filter is to construct what is known in XLSForm jargon as a cascading select, where a selection in a list (for example countries), controls the contents of a second list (for example, States, or regions within that country).  However, as we will see in this article, choice filters can be used well beyond cascading selects.

This blog introduces choice filters in Survey123 and some common techniques to work with them effectively. If you are already familiar with choice filters and cascading selects, you may still want to read this article as it describes a few new interesting things added in version 3.11 (October 2020).

If you are not familiar with choice filters, XLSForms or Survey123 Connect, you may find this content a bit steep. I suggest you watch this 2 minute video with an introduction to XLSForm select questions to warm up. This is not rocket science, but basic familiarity with Survey123 Connect and XLSForms is assumed.

The basics (Cascading selects)

In XLSForm jargon, choice filters are often associated with the notion of 'cascading selects'. The idea is that a selection in one list, determines which values will be presented in the following list. Say we have a list of countries, and a second list of regions. As you select a country in the first list, you want the second list to just show regions for that country.

Below is what the countries and regions cascading select example could look like:

Survey123 Choice Filters

Building your own cascading select is pretty easy. Here is a brief step by step guide using a hyper-simplified countries and regions example:

  • Open Survey123 Connect and create a new survey using the Advanced template
  • Switch to the choices worksheet and add the countries and regions lists as shown below:

  • Now switch to the survey worksheet and add two select_one questions.

  • Save your spreadsheet and preview your form in Survey123 Connect. All the regions will be shown, which is not good. We need a choice filter for that!
  • Switch back to the choices worksheet and add a new country column to the worksheet. This new column can be called whatever you like, but in our case we will call it country.
  • For each of the rows in the regions list, populate the country column accordingly. The values in this column must match the name values from the countries list.

  • Now that the choices worksheet specifies the country for each of the regions, we can define a choice filter in the survey worksheet.
  • Switch to the survey worksheet and look for the choice_filter column. Apply the following expression to the region question: selected(${country},country)

  • Save your spreadsheet and preview your changes in the Survey123 Connect preview. Now your cascading select should work as expected: As you select a country, the list of regions changes accordingly.

As you can see the concept is simple: The expression in the choice_filter column of a select question determines which elements of the list will be displayed. In the choice_filter expression you can reference questions from your survey and  custom columns from the choices worksheet. In our scenario, the expression was selected(${country},country). The choice_filter expression is evaluated against every choice in the region list.  If the expression evaluates to true, then the choice is kept. If the expression evaluates to false, the choice is dropped.

If you have been using choice filters for a while with Survey123, I bet your choice_filter expression would have looked like this: country=${country} instead of selected(${country},country).  Both expressions are correct. We will talk more about the differences between these two expressions, and when to best use one or the other, later in this blog in the 'Working with very large lists'.

Chaining cascading selects

In the previous example we constructed what is known as a cascading select. You can cascade, or chain, as many selects as you like. For example, you can add a cities list next:

In the choices worksheet, note that I added a new region column for the cities list.

In the survey worksheet, I added a new select_one question for the cities and its corresponding choice_filter.

Choice filters on checklists (select_multiple)

You can apply choice filters to select_one and select_multiple questions. In the example below, note that I converted the regions and cities questions into checklists. Say for example, you visited Spain. Then I want users to select all the regions they visited in Spain, and also all the cities visited within these regions.

To do the above, I simply changed the type of questions from select_one to select_multiple.  Since the selected() function works for all select questions (select_one and select_multiple), you do not need to change the choice_filter.

Support for cascading selects on select_multiple questions was added in version 3.11 (October 2020)

More on choice_filter expressions

Given that the choice_filter column can evaluate XLSForm expressions, you can get creative in the way you apply filters to lists.  The key concept to understand is the following:

A choice_filter expression is evaluated against every choice in the list. If the choice_filter expression evaluates to true, then the record is kept in the list. If it evaluates to false, the record is hidden from the list.

With the above in mind, here are some ideas:

  • Filters on numeric values: If the choice list custom column contains numeric values instead of text, the > (greater than), < (less than), = (equal), etc. comparison operators will be useful. Below is an example using price<=${dollars} as a choice filter.  The range question at the beginning sets a numeric value (the dollar amount), which is compared against the price custom column added in the items choice list.

  • Functions on strings: Other than the selected() function and = operator, which we already described, there are other functions you can use to work with strings. Let's pretend we need some work done by a contractor. We want to use a first list to define the work that needs to be done. The list of contractors needs to be filtered to show only the contractors with the necessary skills.

Unlike the countries and regions example we explored earlier, in this case elements from the filtered list can appear multiple times. For example, Patricia is an electrician, gardener and welder. As shown in the screenshot below I added a skills column to the contractors list. This column includes a comma separated list of the skills of each contractor.

To find if a particular skill is present within the skills column, we can use a regular expression such as regex(skills, ${skill}). Alternatively, we can also use contains(skills, ${skill}).  Other string functions such as starts-with() or ends-with() can be handy as well.

  • Working with custom JavaScript functions: In some cases, to model complex logic, you may want to resort to the use of custom JavaScript functions with the pulldata() function.  The key to use your own custom JavaScript functions is simple: if your function returns true, the choice will be added. Otherwise it will be hidden. This option is powerful but it can, depending on the complexity of your function and the length of the list, add an overhead to your app.

As usual, using the survey samples that come with Connect is often the best. In Survey123 Connect, create a new survey and select the Cascading Select sample. You will find in there a few additional great examples illustrating the different patterns you can follow to create filter expressions.

Working with very large lists

Very long lists can slow down your survey.  The definition of a 'long list' varies. It varies depending on the compute power of your device as well as the size of your survey. Generally speaking, you should not worry too much about performance when working with lists under 500 records, but as you get to that threshold, you may want to consider a few tips to optimize lists in your survey.

There are different aspects that require your consideration when working with long lists:

  • How you display lists: Appearance
  • How you store the lists: External choice lists
  • How you filter lists: Choice Filter expressions

Appearance

Survey123 supports multiple ways to display lists. By default, choices in a list are directly presented in the form. In this way, end users can see right away all the choices and tap on them quickly.  This default appearance, is great for very small lists (under 10 elements). For lists that grow larger, this default appearance is not adequate because it forces users to scroll the entire form in order to see elements at the bottom.  If the list grows even larger, the initial loading time of your form will be negatively affected because the default appearance forces Survey123 to render every choice in the list within the form.

Using the XLSForm minimal appearance , you collapse list choices into a dropdown control.  Everyone wins: end users do not need to scroll up and down the entire form, and Survey123 does not need to render the entire list until the user chooses to open the dropdown.   When working with long lists, the minimal appearance reduces the initial survey loading time.

You can also use the autocomplete the appearance. With autocomplete, the dropdown list will not show any choices until the user starts typing. This also saves time because only elements in the list that match the user input need to be rendered. For very long lists, autocomplete is often the best option for both the end user as well as to speed up your survey behavior.

Ultimately, you will need to choose what appearance works best for you. The performance differences of these different options will vary depending on your device too. As I wrote this article, I measured how much time a list with 1,500 elements took to load using the different appearances. Autocomplete was the winner. It reduced loading time 400% when compared to the default appearance and it was also slightly faster than the minimal appearance.

External choice lists

The most common way to define a list in XLSForms is through the choices worksheet. Using the choices worksheet is convenient because you can easily visualize and change your lists when authoring your XLSForm. All it takes is to switch from the survey to the choices worksheet.  Having said this, it is best practice to keep very long lists as separate CSV files.  External CSV files reduce the initial loading time of your survey and accelerate choice filters on your lists. 

Starting with version 3.11, Survey123 supports variations of select_one and select_multiple question types that allow you to reference lists in external files: select_one_from_file and select_multiple_from_file

The Survey123 Connect XLSForm templates do not include the select_one_from_file and select_multiple_from_file question types in the drop-down, but you can add them manually to your XLSForm.

The XLSForm below shows how these new types could be used. The number of municipalities in the Philippines is roughly 1,500. Already a big list, so we are using a select_one_from_file. Note that the list name referenced in this question is referencing a csv file called municipalities.csv. Municipalities are broken down into barangays. As of 2020, there are more than 40,000 barangays in the Philippines. So again, we use a _from_file select to speed up loading time and filtering.  Note that in this case we choose a select_multiple_from_file and also applied a choice filter.

typenamelabelchoice_filter
select_one_from_file municipalities.csvmunicipalityMunicipalities
select_multiple_from_file barangays.csvbarangaysBarangaysmunicipality=${municipality}

The contents of the csv files are not much different from what is included in the standard XLSForm choices worksheet. The list_name column is not needed, as the name of the list is defined by the csv file name (no spaces allowed in the name). The name and label columns for choices in your list are mandatory and any other column (language columns, media, image, etc) are optional.

The csv file names do not accept spaces and the file must be stored within the media folder of your survey.

I want to emphasize here that using external csv files for your lists is adequate for very long lists (over 500 choices), but it is not going to necessarily add much, other than overhead to you, for smaller lists. Keeping lists in the choices worksheet of your XLSForm is a very valid and convenient approach.  For most of your forms, you will want to keep your list definitions in there.  It is only when you work with monster lists that _from_file selects will help you.

The use of _from_file selects supports both select_one questions (select_one_from_file) and select_multiple (select_multiple_from_file).  You can also apply choice filters, calculations, constraints and any other XLSForm tricks you already.  Other than where the list is stored (external file versus the choices worksheet in your XLSForm), your XLSForm authoring experience should remain the same.

The XLSForm specification supports a variation of external choice lists called external selects. I will not be covering in this blog how to work with external selects, but if you are interested you can have a look at them in the Cascading and external selects—ArcGIS Survey123 | Documentation help topic.  I personally like to use external choice lists instead of external selects.

Choice filter expressions

How you write your choice filter expressions can impact performance as well. The fastest choice filters are those using simple comparison operators. For example:

  • country=${country}
  • dollars>${price}
  • code!=${code}
  • priority>=${priority_reported}

Remember that comparison operators like the above work only against select_one questions. For select_multiple questions you need to go with functions like selected(), regex(), contains() and others, which can't be optimized by Survey123 as efficiently as simple comparison operators.

You should not be afraid of using selected(), regex() and other functions as performance will generally be pretty good, but if you want to squeeze the best of your device, simple operators will help.

Be cautious when using custom JavaScript functions with the pulldata() function in choice filters. Since your JavaScript function needs to be evaluated once for every record in the list, performance can be greatly affected. 

Learning more

As you can see, there is a lot around choice filters in Survey123. If you would like to see more examples of expressions you can apply, check the Cascading Selects sample included in Connect. 

more
9 26 5,315
IsmaelChivite
Esri Frequent Contributor

The Survey123 web app supports a variety of URL parameters that can be used to initialize the contents, look and feel, and behavior of your online surveys. This article describes how you can make the use of URL parameters more secure, by encoding them.

In the following example, URL parameters are used to prepopulate and hide the submittedBy question:

https://survey123.arcgis.com/share/1cb28b212b5542acbbdbaa35feba0765?field:submittedBy=Fernando%20Par...

Since the URL parameters are in clear text, a smart user could look at the web browser's navigation bar, figure out that a question is being hidden, and manipulate the URL parameters to show and change the question.

Here is the same URL, with encoded parameters:

https://survey123.arcgis.com/share/1cb28b212b5542acbbdbaa35feba0765?code=ShFmaWVsZDpzdWJtaXR0ZWRCeXI...

Once the URL parameters are encoded, only the Survey123 web app can read them, making the URL much more secure.

Tip: If you are not familiar with Survey123 web app URL parameters, check these two blog posts:

Introduction to URL parameter encoding in Survey123

URL parameter encoding is a technique that allows you to obscure the contents of URL parameters. When parameters are encoded, it is not possible for end users to manipulate existing or add new parameters to the URL. This functionality is only available with Survey123 online forms published with version 3.11 or newer.

Tip: If you wish to upgrade your online forms to newer versions, see:

Encoding parameters manually (&encodeUrlParams=true)

Let's pretend you want to add a link to your survey in a website and Facebook. Using URL parameters, you prepopulate a hidden question so you can tell from which source (website or Facebook) the survey was sent. In this case, since you only need to create and encode two links, you can simply use your web browser. Here is how:

  • Create your own online survey URL and add parameters to it as usual.
  • Add &encodeUrlParams=true at the very end of the URL and reload your page.
  • The encoded URL will appear in your web browser's navigation bar.
  • Copy the URL and use it somewhere else.

This animation shows the process step by step.


Encoding parameters with the Survey123 REST API

Manually encoding your survey URLs is only practical when you have a handful of URLs to share. If you need to create many Survey123 links, or you need to create them dynamically, the best choice is to use the Survey123 REST API to programmatically encode your URL parameters.

The encodeUrlParams operation on the Survey123 REST API allows you to pass a collection of Survey123 parameters and get back the encoded URL string.

The URL endpoint of this operation is: https://survey123.arcgis.com/api/encodeUrlParams

It expects a POST request including a valid ArcGIS token and the parameters you want to encode. For example:

  {
"token": "<TOKEN>",
"params": {
        "field:submittedBy": "Fernando Paredes",
        "hide": ["field: submittedBy"]
}
"portalUrl": "<Portal URL>",  //Optional

  }

Note that it is not necessary to pass any information about your survey. All you need to pass are the URL parameters that you wish to encode and a token. The portalUrl parameter is only needed if you are working against ArcGIS Enterprise.

 

The response from the Survey123 REST API will look like the following:

{
"code": "ShBmaWVsZDpjdXNUcmFja05vchRmaWVsZDpjdXNUcmFja05vPTEyMw==",
    "success": true
}

The value of the code property represents your URL parameters, encoded.

Encoding parameters in bulk with the Survey123 REST API and Python

Say you wish to send an email to a very large group of people. You want the email to include a personalized Survey123 link, prepopulating a few questions in the survey based on the recipient's information you already have. You need to create a CSV file that you can import into an email marketing tool like MailChimp, Drip or MailerLite, including the email of the recipients and their corresponding Survey123 link. Of course, you want the Survey123 links to have the URL parameters encoded.

You can use the encodeUrlParams operation in the Survey123 REST API from any scripting language able to send a POST request. The example below highlights how links can be created from an ArcGIS Notebook. The code first performs a query on a feature service and generates a unique link for each record in the returned set. The script prints out the email and its corresponding Survey123 link with encoded parameters. Click here to preview the notebook.

Survey123 Notebook

As you can see, the code is straight-forward. A single POST request lets you specify the URL parameters you want to encode. The output of the Survey123 REST API can then be added to your Survey123 link.

Encoding parameters with the Survey123 REST API and Microsoft Power Automate

 

If you want to create a collection of links in bulk from some information you already have, the Python approach described above is a good fit. In other cases, you may need to create links dynamically.

For example, say you publish a survey for people to self-report damage to their property after a natural disaster. You would like respondents to automatically receive an email with a link that would allow them to update their own information. As you already know, using URL parameters it is possible to create a survey link to open a record in edit mode. You can also dynamically create such a link and email it right after a survey is submitted using Microsoft Power Automate and Integromat, but is it possible to also encode the URL parameters? Yes!

Overall, the Microsoft Power Automate flow could look like the following:

Survey123 Automate

Note that the trigger for the flow is the Survey123 connector. Every time a survey is submitted, the flow is executed. The payload of the trigger includes the globalId of the record just created, which is made available in Microsoft Power Automate as dynamic content.

The HTTP module is used to encode the URL parameters.

The HTTP method is set to POST. The URI parameter targets the encodeUrlParams operation in the Survey123 REST API. In the queries group, the token parameter is populated dynamically with the token value provided in the Survey123 webhook payload. The params parameter includes the URL parameters we want encoded. In this case, we pass edit for the mode and the globalId of the feature we want to edit. The globalId is coming as dynamic content from the Survey123 webhook payload as well.

 

Next the Parse JSON module is used, so that later we can easily extract the encoded values from the output of the Survey123 REST API.

Finally, the Send Email module is used to dynamically create a Survey123 link within the subject of the email. Note that the encoded parameters are dynamically added to the Survey123 link.

Summary

The Survey123 web app supports several URL parameters, allowing you to preload responses to questions, and change the look and feel and behavior of your online surveys. Using the encodeUrlParams operation in the Survey123 REST API, you can obscure these parameters to avoid manipulation by end users. 

You can generate Survey123 links with encoded parameters manually right from your browser, programmatically through scripting languages like Python, and dynamically through workflow automation in Microsoft Power Automate and Integromat.

more
2 10 2,026
IsmaelChivite
Esri Frequent Contributor

The ranking question lets people sort items in a list in order of preference. Here is an example where we ask respondents to rank a series of topics for an upcoming Survey123 workshop. Note how the ranking of a topic in the list can be changed by simply dragging it to the desired position. Users promote their favorite topics to the top of the list.

As the user submits the survey, choices in the ranking question are scored accordingly to their position in the list: The higher in the list, the higher the score. Survey123 lets you analyze survey responses to understand the average and variance of the scores for each item. The chart below shows preferences showed by respondents. The "Survey123 design best practices" ranked first with an average score of 6.53, followed by the "Survey123 reports" topic.

The Ranking question is a great resource anytime you want to measure people's preferences over a list of well defined choices.

This blog post describes how you can create your own Ranking questions with Survey123. It first presents how you can add this type of question using Survey123 designer. It then shows how you can explore results from the Survey123 website and ultimately, how you can use XLSForms to use this question type in Survey123 Connect.

Adding a Ranking question in the Survey123 designer

The Survey123 web designer lets you visually build your own surveys from a web browser. To add a Ranking question to your survey, drag it into the design preview. You can interactively add choices to the list, or add them in bulk as shown in the animation below.

Adding too many choices to a Ranking question will be overwhelming to users. It is recommended that you do not add more than 6 choices.

Choice randomization

It's human nature that people tend to either favor whatever choices are presented first (primacy effect) or last (recency effect). To avoid biased responses, you may want to randomize how choices are initially presented to the user. In Survey123, you can choose to randomize choices in all types of lists (dropdown, choice lists, checklists...). Randomization is particularly useful for the ranking question.  In the Survey123 web designer, you will find a 'Show choices in random order' option. It will be enabled by default when you add a new Ranking question, but you can choose to disable it.

Analyzing rank results

As the user submits the survey, choices in the Ranking question are scored. Here is how scoring works: If you have 5 choices in your list, the choice at the top gets a score of 5, the choice after that a 4, etc. If you have a list with 7 choices, the one at the top gets a score of 7... You get the point.

When you look at the results of your survey in the Survey123 website, you will be presented with a chart and a table. The chart sorts your choices using the average score. It also shows you the number of responses, and the number of times the question has been skipped (if appropriate). Here is an example. This chart will give you insight as to who are the winners, overall. 

The table is useful to understand the variance of the rankings. The table indicates what the average score is, and how many times a particular choice was ranked at each position.  For example, in the table below we see that the 'Survey123 design best practices' topic was selected as the top choice by 83 of the respondents (almost 30%).  We also see that while 'XLSForm advanced techniques' ranked third position, it was chosen as the first choice by a larger group than 'Survey123 reports', which ranks second.  This tells us that on average, the Survey123 reports topic is more popular than 'Advanced Techniques', but proponents of 'Advanced Techniques' care proportionally more about the topic.

If you were to look at the raw data stored in your GIS records, the output of a rank question is a comma separated list of values.  The first value in the list is the top choice for that record, followed in descending order by the rest.

Ranking questions in Survey123 Connect

If you work with Survey123 Connect, you can also add ranking questions. Use the rank question type and assign it a list from your choices worksheet. The syntax is similar to that of select_one and select_multiple questions, but you use rank instead.  If you want to randomize the choices, set the parameters value to randomize=true. It could look something like this:

typenamelabelparameters
rank topicstopics_rankedFavorite topicsrandomize=true

Ranked choices are stored as a comma separated list of choice names.

banana, apple, orange, kiwi

The comma-separated list of values honors the order of the user selection. In the example above, banana was placed at the top, followed by apple, orange and kiwi.

Keeping this in mind, you can do a few clever things with XLSForms. A common request is to be able to store the score of each choice as an attribute. Here is what I mean. Note how the scores for the fruits automatically calculate as the choices are ordered:

Once you know how to get the score of an item, you can use these values to implement your own custom validation or skip logic.

Getting the score values can be accomplished with the help of a simple JavaScript function and pulldata. If you are not familiar with using custom JavaScript functions with XLSForms, seehttps://community.esri.com/groups/survey123/blog/2020/08/07/extending-survey123-smart-forms-with-cus... 

Here is what the XLSForm could look like:

typenamelabelcalculationparameters
rank fruitsfruits_rkndFavorite fruitsrandomize=true
calculatebananaBananapulldata("@javascript","myJS.js","getScore","banana",${fruits_rnkd})
calculateappleApplepulldata("@javascript","myJS.js","getScore","apple",${fruits_rnkd})
calculatekiwiKiwipulldata("@javascript","myJS.js","getScore","kiwi",${fruits_rnkd})

And here is a JavaScript function to back it up:

function getScore (fruit, ranked_fruits)

{

         var fruits = ranked_fruits.split(',');
         return fruits.indexOf(fruit) + 1;

}

more
1 5 1,242