Survey123 Tricks of the Trade: Ranking question and choice randomization

7525
14
10-16-2020 09:50 AM
IsmaelChivite
Esri Notable Contributor
1 14 7,525

The ranking question lets people sort items in a list in order of preference. Here is an example where we ask respondents to rank a series of topics for an upcoming Survey123 workshop. Note how the ranking of a topic in the list can be changed by simply dragging it to the desired position. Users promote their favorite topics to the top of the list.

As the user submits the survey, choices in the ranking question are scored accordingly to their position in the list: The higher in the list, the higher the score. Survey123 lets you analyze survey responses to understand the average and variance of the scores for each item. The chart below shows preferences showed by respondents. The "Survey123 design best practices" ranked first with an average score of 6.53, followed by the "Survey123 reports" topic.

The Ranking question is a great resource anytime you want to measure people's preferences over a list of well defined choices.

This blog post describes how you can create your own Ranking questions with Survey123. It first presents how you can add this type of question using Survey123 designer. It then shows how you can explore results from the Survey123 website and ultimately, how you can use XLSForms to use this question type in Survey123 Connect.

Adding a Ranking question in the Survey123 designer

The Survey123 web designer lets you visually build your own surveys from a web browser. To add a Ranking question to your survey, drag it into the design preview. You can interactively add choices to the list, or add them in bulk as shown in the animation below.

Adding too many choices to a Ranking question will be overwhelming to users. It is recommended that you do not add more than 6 choices.

Choice randomization

It's human nature that people tend to either favor whatever choices are presented first (primacy effect) or last (recency effect). To avoid biased responses, you may want to randomize how choices are initially presented to the user. In Survey123, you can choose to randomize choices in all types of lists (dropdown, choice lists, checklists...). Randomization is particularly useful for the ranking question.  In the Survey123 web designer, you will find a 'Show choices in random order' option. It will be enabled by default when you add a new Ranking question, but you can choose to disable it.

Analyzing rank results

As the user submits the survey, choices in the Ranking question are scored. Here is how scoring works: If you have 5 choices in your list, the choice at the top gets a score of 5, the choice after that a 4, etc. If you have a list with 7 choices, the one at the top gets a score of 7... You get the point.

When you look at the results of your survey in the Survey123 website, you will be presented with a chart and a table. The chart sorts your choices using the average score. It also shows you the number of responses, and the number of times the question has been skipped (if appropriate). Here is an example. This chart will give you insight as to who are the winners, overall. 

The table is useful to understand the variance of the rankings. The table indicates what the average score is, and how many times a particular choice was ranked at each position.  For example, in the table below we see that the 'Survey123 design best practices' topic was selected as the top choice by 83 of the respondents (almost 30%).  We also see that while 'XLSForm advanced techniques' ranked third position, it was chosen as the first choice by a larger group than 'Survey123 reports', which ranks second.  This tells us that on average, the Survey123 reports topic is more popular than 'Advanced Techniques', but proponents of 'Advanced Techniques' care proportionally more about the topic.

If you were to look at the raw data stored in your GIS records, the output of a rank question is a comma separated list of values.  The first value in the list is the top choice for that record, followed in descending order by the rest.

Ranking questions in Survey123 Connect

If you work with Survey123 Connect, you can also add ranking questions. Use the rank question type and assign it a list from your choices worksheet. The syntax is similar to that of select_one and select_multiple questions, but you use rank instead.  If you want to randomize the choices, set the parameters value to randomize=true. It could look something like this:

typenamelabelparameters
rank topicstopics_rankedFavorite topicsrandomize=true

Ranked choices are stored as a comma separated list of choice names.

banana, apple, orange, kiwi

The comma-separated list of values honors the order of the user selection. In the example above, banana was placed at the top, followed by apple, orange and kiwi.

Keeping this in mind, you can do a few clever things with XLSForms. A common request is to be able to store the score of each choice as an attribute. Here is what I mean. Note how the scores for the fruits automatically calculate as the choices are ordered:

Once you know how to get the score of an item, you can use these values to implement your own custom validation or skip logic.

Getting the score values can be accomplished with the help of a simple JavaScript function and pulldata. If you are not familiar with using custom JavaScript functions with XLSForms, seehttps://community.esri.com/groups/survey123/blog/2020/08/07/extending-survey123-smart-forms-with-cus... 

Here is what the XLSForm could look like:

typenamelabelcalculationparameters
rank fruitsfruits_rkndFavorite fruitsrandomize=true
calculatebananaBananapulldata("@javascript","myJS.js","getScore","banana",${fruits_rnkd})
calculateappleApplepulldata("@javascript","myJS.js","getScore","apple",${fruits_rnkd})
calculatekiwiKiwipulldata("@javascript","myJS.js","getScore","kiwi",${fruits_rnkd})

And here is a JavaScript function to back it up:

function getScore (fruit, ranked_fruits)

{

         var fruits = ranked_fruits.split(',');
         return fruits.indexOf(fruit) + 1;

}

14 Comments
surveysmanager
New Contributor

Hi @IsmaelChivite , this advanced option by JavaScript seems very powerful! I'm not familiar with coding, I tried to follow your instructions and the JavaScript function you suggest here, and of course I missed something because it doesn't work.. at any position is my choice, I always get 0 as score.

Actually the function I'm trying to reach is slightly different from the one you illustrated and I think I do not need to use the JavaScript because it is a pretty basic goal.

I need just to extract the first three items from the comma separated string produced from a rank question type in three different fields. I found somewhere in the blog and posts that this result can be reached by the selected-at  function.  It works indeed well when I submit the survey by the web form but it doesn't' work at all if I submit it by the field app.

I cannot find any posts related to this anomaly and I need the user to be able to submit by both web and field app. 

Any help would be really appreciated!

Thank you

Elisabetta

IsmaelChivite
Esri Notable Contributor

Hi @surveysmanager   I was able to reproduce the issue you describe: the selected-at function does not work against rank questions in the Survey123 field app.  I will add an internal note for the team to be aware of this but I strongly suggest you open an Esri Technical Support case so this software defect is officially documented.

I agree that for your particular case, using selected-at looks like a cleaner solution that a custom JS function. However, to get things going while this issue is addressed, you may want to get help from our Esri Tech Support team to get your JS function in place.  Thanks!

IsmaelChivite
Esri Notable Contributor

Hi Again. The selected-at function BUG will be addressed in Survey123 version 3.12 (tentatively planned for release in early 2021).

MTI
by
New Contributor II

Hello - I'd like to provide a choice/field in the ranking question that could provide an "other" comment or where the respondent can type in their choice.  Is this possible?  TX!

IsmaelChivite
Esri Notable Contributor

@MTINot possible. Sorry!

by Anonymous User
Not applicable

Hi, I am trying to use a question from Survey123 web app in Dashboards and using Arcade to try and replicate a list of results or serial charts like appears in the Survey123 web design.  I know that the multiple-choice question uses Javascript, but is there a way to get that look in Dashboards?

jswiggy_0-1616768255473.png

 

However, when I look at it in Dashboards, the values are reported as a mix.

jswiggy_1-1616768316059.png

Rather than being separated out individually, they are lumped as a unique selection, when what I want is each value separate, like in the chart.  Any ideas of how this might be accomplished?

Thanks!

JerrySneary
New Contributor III

Hi @IsmaelChivite 

My question is the same as @Anonymous User above. I also want to create a dashboard for the public and there is no way that I can tell to do this in Dashboard, Arcade is not allowed on the Serial Chart Widget. Am I mistaken? Or can you add numeric values within the form based on where the applicant ranks that info?

Thanks,

Jerry

mettegreve
New Contributor III

I have a select_one question with randomize=true. Built using Survey123 Connect and users must answer online, so no Survey123 App. It is published to our portal.

BUT: my radiobuttons disappear, when I use randomize=true. They come back, when I delete it:

With randomize=true:

mettegreve_0-1643368577980.png

 

 

Without:

mettegreve_1-1643368577983.png

 

ArmstKP
Occasional Contributor III

@IsmaelChivite In your JavaScript above, you mention:  ranked_fruits. I do not see where it is referenced or mentioned in any of your screenshots?

 

function getScore (fruit, ranked_fruits)

{

         var fruits = ranked_fruits.split(',');
         return fruits.indexOf(fruit) + 1;

}

IsmaelChivite
Esri Notable Contributor

IsmaelChivite_0-1651268312798.png

 

rankedFruits is the name of the second parameter in the JS function.

Note that this parameter is fed with the fruits_rnkd question

In the blog above you need to horizontally scroll the table to see the whole thing. For your convenience, I pasted an image in this response.

OscarUrbinaLeal
New Contributor II

I have a question, I did a similar example in two questions with some buildings, but when I test the answer (of four possible answers), it counts each answer as 1. Shouldn't it count the answers as 1, 2, 3, and 4, depending on their rank? 

This is what I used in excel and Java:

This is the ranking #1 I did:

rank scen2_1rankscen2_1Please rank which building should the authorities combat first

and this is the formula I used in the calculation field:

pulldata("@javascript","myJS.js","getScore","Church_1",${rankscen2_1})

And in the Java Script named myJS.js I used this:

function getScore (building, rankscen2_1)

{

var building = rankscen2_1.split(',');

return building.indexOf(building) + 1;

}

Could you help me telling me if its normal that it shows only as one each answer, or how could I fix it?

Thanks a lot

RyanBohan
Occasional Contributor III

It seems the rank question does not like spaces in the name.  I was getting an error on the webform that would not let me complete this question.  Removing the space(s) seems to do the trick

RyanBohan_0-1684358865415.png

 

ZachBodenner
MVP Regular Contributor

I have another use case for ranked questions: What if I have a list of ranked questions where there is a correct order? Say I have a list like, "Which numbers are the biggest?" 10,7,5,3,1. I want my users to put them in descending order, and obviously there is a correct way to do it (not my real question, but you get the drift). Now, how could I assess the accuracy of their ranking using some kind of hidden-question calculation? Something like, if 10 is in the top position, it's worth 5 points, on down the line. But if someone put one at the top, and then the rest of the numbers in correct order, maybe they should get some points for being mostly right? Any ideas for how I might calculate that out?

AJ_devaccount
Occasional Contributor

@IsmaelChivite is there a way to get the scores of the list items without JS? I have a public map, so unfortunately I can't use JS to get the scores of each item. Thank you for your help!