Survey123 Tricks of the Trade: Choice Filters

[IsmaelChivite]

The best way to tackle a big problem is to break it down into smaller ones. Lists are no different. Choice filters let you leverage user-entered values in your form to dynamically control which elements of a list are shown or hidden. The most common use for a choice filter is to construct what is known in XLSForm jargon as a cascading select, where a selection in a list (for example countries), controls the contents of a second list (for example, States, or regions within that country).  However, as we will see in this article, choice filters can be used well beyond cascading selects.

This blog introduces choice filters in Survey123 and some common techniques to work with them effectively. If you are already familiar with choice filters and cascading selects, you may still want to read this article as it describes a few new interesting things added in version 3.11 (October 2020).

If you are not familiar with choice filters, XLSForms or Survey123 Connect, you may find this content a bit steep. I suggest you watch this 2 minute video with an introduction to XLSForm select questions to warm up. This is not rocket science, but basic familiarity with Survey123 Connect and XLSForms is assumed.

The basics (Cascading selects)

In XLSForm jargon, choice filters are often associated with the notion of 'cascading selects'. The idea is that a selection in one list, determines which values will be presented in the following list. Say we have a list of countries, and a second list of regions. As you select a country in the first list, you want the second list to just show regions for that country.

Below is what the countries and regions cascading select example could look like:

Building your own cascading select is pretty easy. Here is a brief step by step guide using a hyper-simplified countries and regions example:

• Open Survey123 Connect and create a new survey using the Advanced template
• Switch to the choices worksheet and add the countries and regions lists as shown below:

• Now switch to the survey worksheet and add two select_one questions.

• Save your spreadsheet and preview your form in Survey123 Connect. All the regions will be shown, which is not good. We need a choice filter for that!
• Switch back to the choices worksheet and add a new country column to the worksheet. This new column can be called whatever you like, but in our case we will call it country.
• For each of the rows in the regions list, populate the country column accordingly. The values in this column must match the name values from the countries list.

• Now that the choices worksheet specifies the country for each of the regions, we can define a choice filter in the survey worksheet.
• Switch to the survey worksheet and look for the choice_filter column. Apply the following expression to the region question: selected(${country},country)

• Save your spreadsheet and preview your changes in the Survey123 Connect preview. Now your cascading select should work as expected: As you select a country, the list of regions changes accordingly.

As you can see the concept is simple: The expression in the choice_filter column of a select question determines which elements of the list will be displayed. In the choice_filter expression you can reference questions from your survey and  custom columns from the choices worksheet. In our scenario, the expression was selected(${country},country). The choice_filter expression is evaluated against every choice in the region list.  If the expression evaluates to true, then the choice is kept. If the expression evaluates to false, the choice is dropped.

If you have been using choice filters for a while with Survey123, I bet your choice_filter expression would have looked like this: country=${country} instead of selected(${country},country).  Both expressions are correct. We will talk more about the differences between these two expressions, and when to best use one or the other, later in this blog in the 'Working with very large lists'.

Chaining cascading selects

In the previous example we constructed what is known as a cascading select. You can cascade, or chain, as many selects as you like. For example, you can add a cities list next:

In the choices worksheet, note that I added a new region column for the cities list.

In the survey worksheet, I added a new select_one question for the cities and its corresponding choice_filter.

Choice filters on checklists (select_multiple)

You can apply choice filters to select_one and select_multiple questions. In the example below, note that I converted the regions and cities questions into checklists. Say for example, you visited Spain. Then I want users to select all the regions they visited in Spain, and also all the cities visited within these regions.

To do the above, I simply changed the type of questions from select_one to select_multiple.  Since the selected() function works for all select questions (select_one and select_multiple), you do not need to change the choice_filter.

Support for cascading selects on select_multiple questions was added in version 3.11 (October 2020)
           

More on choice_filter expressions

Given that the choice_filter column can evaluate XLSForm expressions, you can get creative in the way you apply filters to lists.  The key concept to understand is the following:

A choice_filter expression is evaluated against every choice in the list. If the choice_filter expression evaluates to true, then the record is kept in the list. If it evaluates to false, the record is hidden from the list.

With the above in mind, here are some ideas:

• Filters on numeric values: If the choice list custom column contains numeric values instead of text, the > (greater than), < (less than), = (equal), etc. comparison operators will be useful. Below is an example using price<=${dollars} as a choice filter.  The range question at the beginning sets a numeric value (the dollar amount), which is compared against the price custom column added in the items choice list.

• Functions on strings: Other than the selected() function and = operator, which we already described, there are other functions you can use to work with strings. Let's pretend we need some work done by a contractor. We want to use a first list to define the work that needs to be done. The list of contractors needs to be filtered to show only the contractors with the necessary skills.

Unlike the countries and regions example we explored earlier, in this case elements from the filtered list can appear multiple times. For example, Patricia is an electrician, gardener and welder. As shown in the screenshot below I added a skills column to the contractors list. This column includes a comma separated list of the skills of each contractor.

To find if a particular skill is present within the skills column, we can use a regular expression such as regex(skills, ${skill}). Alternatively, we can also use contains(skills, ${skill}).  Other string functions such as starts-with() or ends-with() can be handy as well.

• Working with custom JavaScript functions: In some cases, to model complex logic, you may want to resort to the use of custom JavaScript functions with the pulldata() function.  The key to use your own custom JavaScript functions is simple: if your function returns true, the choice will be added. Otherwise it will be hidden. This option is powerful but it can, depending on the complexity of your function and the length of the list, add an overhead to your app.

As usual, using the survey samples that come with Connect is often the best. In Survey123 Connect, create a new survey and select the Cascading Select sample. You will find in there a few additional great examples illustrating the different patterns you can follow to create filter expressions.

Working with very large lists

Very long lists can slow down your survey.  The definition of a 'long list' varies. It varies depending on the compute power of your device as well as the size of your survey. Generally speaking, you should not worry too much about performance when working with lists under 500 records, but as you get to that threshold, you may want to consider a few tips to optimize lists in your survey.

There are different aspects that require your consideration when working with long lists:

• How you display lists: Appearance
• How you store the lists: External choice lists
• How you filter lists: Choice Filter expressions

Appearance

Survey123 supports multiple ways to display lists. By default, choices in a list are directly presented in the form. In this way, end users can see right away all the choices and tap on them quickly.  This default appearance, is great for very small lists (under 10 elements). For lists that grow larger, this default appearance is not adequate because it forces users to scroll the entire form in order to see elements at the bottom.  If the list grows even larger, the initial loading time of your form will be negatively affected because the default appearance forces Survey123 to render every choice in the list within the form.

Using the XLSForm minimal appearance , you collapse list choices into a dropdown control.  Everyone wins: end users do not need to scroll up and down the entire form, and Survey123 does not need to render the entire list until the user chooses to open the dropdown.   When working with long lists, the minimal appearance reduces the initial survey loading time.

You can also use the autocomplete the appearance. With autocomplete, the dropdown list will not show any choices until the user starts typing. This also saves time because only elements in the list that match the user input need to be rendered. For very long lists, autocomplete is often the best option for both the end user as well as to speed up your survey behavior.

Ultimately, you will need to choose what appearance works best for you. The performance differences of these different options will vary depending on your device too. As I wrote this article, I measured how much time a list with 1,500 elements took to load using the different appearances. Autocomplete was the winner. It reduced loading time 400% when compared to the default appearance and it was also slightly faster than the minimal appearance.

External choice lists

The most common way to define a list in XLSForms is through the choices worksheet. Using the choices worksheet is convenient because you can easily visualize and change your lists when authoring your XLSForm. All it takes is to switch from the survey to the choices worksheet.  Having said this, it is best practice to keep very long lists as separate CSV files.  External CSV files reduce the initial loading time of your survey and accelerate choice filters on your lists. 

Starting with version 3.11, Survey123 supports variations of select_one and select_multiple question types that allow you to reference lists in external files: select_one_from_file and select_multiple_from_file. 

The Survey123 Connect XLSForm templates do not include the select_one_from_file and select_multiple_from_file question types in the drop-down, but you can add them manually to your XLSForm.

The XLSForm below shows how these new types could be used. The number of municipalities in the Philippines is roughly 1,500. Already a big list, so we are using a select_one_from_file. Note that the list name referenced in this question is referencing a csv file called municipalities.csv. Municipalities are broken down into barangays. As of 2020, there are more than 40,000 barangays in the Philippines. So again, we use a _from_file select to speed up loading time and filtering.  Note that in this case we choose a select_multiple_from_file and also applied a choice filter.

type	name	label	choice_filter
select_one_from_file municipalities.csv	municipality	Municipalities
select_multiple_from_file barangays.csv	barangays	Barangays	municipality=${municipality}

The contents of the csv files are not much different from what is included in the standard XLSForm choices worksheet. The list_name column is not needed, as the name of the list is defined by the csv file name (no spaces allowed in the name). The name and label columns for choices in your list are mandatory and any other column (language columns, media, image, etc) are optional.

The csv file names do not accept spaces and the file must be stored within the media folder of your survey.
         

I want to emphasize here that using external csv files for your lists is adequate for very long lists (over 500 choices), but it is not going to necessarily add much, other than overhead to you, for smaller lists. Keeping lists in the choices worksheet of your XLSForm is a very valid and convenient approach.  For most of your forms, you will want to keep your list definitions in there.  It is only when you work with monster lists that _from_file selects will help you.

The use of _from_file selects supports both select_one questions (select_one_from_file) and select_multiple (select_multiple_from_file).  You can also apply choice filters, calculations, constraints and any other XLSForm tricks you already.  Other than where the list is stored (external file versus the choices worksheet in your XLSForm), your XLSForm authoring experience should remain the same.

The XLSForm specification supports a variation of external choice lists called external selects. I will not be covering in this blog how to work with external selects, but if you are interested you can have a look at them in the Cascading and external selects—ArcGIS Survey123 | Documentation help topic.  I personally like to use external choice lists instead of external selects.

Choice filter expressions

How you write your choice filter expressions can impact performance as well. The fastest choice filters are those using simple comparison operators. For example:

• country=${country}
• dollars>${price}
• code!=${code}
• priority>=${priority_reported}

Remember that comparison operators like the above work only against select_one questions. For select_multiple questions you need to go with functions like selected(), regex(), contains() and others, which can't be optimized by Survey123 as efficiently as simple comparison operators.

You should not be afraid of using selected(), regex() and other functions as performance will generally be pretty good, but if you want to squeeze the best of your device, simple operators will help.

Be cautious when using custom JavaScript functions with the pulldata() function in choice filters. Since your JavaScript function needs to be evaluated once for every record in the list, performance can be greatly affected. 

Learning more

As you can see, there is a lot around choice filters in Survey123. If you would like to see more examples of expressions you can apply, check the Cascading Selects sample included in Connect.