Survey123 Tricks of the Trade: Encode URL parameters in the web app

9674
13
10-16-2020 10:07 AM
IsmaelChivite
Esri Notable Contributor
3 13 9,674

 

The Survey123 web app supports a variety of URL parameters that can be used to initialize the contents, look and feel, and behavior of your online surveys. This article describes how you can make the use of URL parameters more secure, by encrypting them.

 

In the following example, URL parameters are used to prepopulate and hide the submittedBy question:

https://survey123.arcgis.com/share/1cb28b212b5542acbbdbaa35feba0765?field:submittedBy=Fernando%20Par...

Since the URL parameters are in clear text, a smart user could look at the web browser's navigation bar, figure out that a question is being hidden, and manipulate the URL parameters to show and change the question.

Here is the same URL, with encrypted parameters:

https://survey123.arcgis.com/share/1cb28b212b5542acbbdbaa35feba0765?code=ShFmaWVsZDpzdWJtaXR0ZWRCeXI...

Once the URL parameters are encrypted, only the Survey123 web app can read them, making the content in your URL secure.

 

Introduction to URL parameter encoding (encryption) in Survey123

 

URL parameter encoding is a technique that allows you to encrypt the contents of URL parameters. When parameters are encrypted, it is not possible for end users to manipulate existing or add new parameters to the URL. This functionality is only available with Survey123 online forms published with version 3.11 or newer.

 

Encoding parameters manually (&encodeUrlParams=true)

 

Let's pretend you want to add a link to your survey in a website. Using URL parameters, you prepopulate a hidden question so you can tell from which source (website, Facebook, etc) the survey was sent. In this case, since you only need to create and encode two links, you can simply use your web browser. Here is how:

 

  • Create your own online survey URL and add parameters to it as usual.
  • Add &encodeUrlParams=true at the very end of the URL and reload your page.
  • The encoded URL will appear in your web browser's navigation bar.
  • Copy the URL and use it somewhere else.

 

This animation shows the process step by step.

 


 

Encoding parameters with the Survey123 REST API

 

Manually encoding your survey URLs is only practical when you have a handful of URLs to share. If you need to create many Survey123 links, or you need to create them dynamically, the best choice is to use the Survey123 REST API to programmatically encode your URL parameters.

The encodeUrlParams operation on the Survey123 REST API allows you to pass a collection of Survey123 parameters and get back the encoded URL string.

The URL endpoint of this operation is: https://survey123.arcgis.com/api/encodeUrlParams

It expects a POST request including a valid ArcGIS token and the parameters you want to encode. For example:

 

  {
"token": "<TOKEN>",
"params": {
        "field:submittedBy": "Fernando Paredes",
        "hide": ["field: submittedBy"]
}
"portalUrl": "<Portal URL>",  //Optional

  }

Note that it is not necessary to pass any information about your survey. All you need to pass are the URL parameters that you wish to encode and a token. The portalUrl parameter is only needed if you are working against ArcGIS Enterprise.

 

The response from the Survey123 REST API will look like the following:

 

{
"code": "ShBmaWVsZDpjdXNUcmFja05vchRmaWVsZDpjdXNUcmFja05vPTEyMw==",
    "success": true
}

 

The value of the code property represents your URL parameters, encoded.

 

Encoding parameters in bulk with the Survey123 REST API and Python

 

Say you wish to send an email to a very large group of people. You want the email to include a personalized Survey123 link, prepopulating a few questions in the survey based on the recipient's information you already have. You need to create a CSV file that you can import into an email marketing tool like MailChimp, Drip or MailerLite, including the email of the recipients and their corresponding Survey123 link. Of course, you want the Survey123 links to have the URL parameters encoded.

 

You can use the encodeUrlParams operation in the Survey123 REST API from any scripting language able to send a POST request. The example below highlights how links can be created from an ArcGIS Notebook. The code first performs a query on a feature service and generates a unique link for each record in the returned set. The script prints out the email and its corresponding Survey123 link with encoded parameters. Click here to preview the notebook.

 

Survey123 Notebook

 

As you can see, the code is straight-forward. A single POST request lets you specify the URL parameters you want to encode. The output of the Survey123 REST API can then be added to your Survey123 link.

 

Encoding parameters with the Survey123 REST API and Microsoft Power Automate

 

If you want to create a collection of links in bulk from some information you already have, the Python approach described above is a good fit. In other cases, you may need to create links dynamically.

For example, say you publish a survey for people to self-report damage to their property after a natural disaster. You would like respondents to automatically receive an email with a link that would allow them to update their own information. As you already know, using URL parameters it is possible to create a survey link to open a record in edit mode. You can also dynamically create such a link and email it right after a survey is submitted using Microsoft Power Automate and Integromat, but is it possible to also encode the URL parameters? Yes!

 

Overall, the Microsoft Power Automate flow could look like the following:

 

Survey123 Automate

 

Note that the trigger for the flow is the Survey123 connector. Every time a survey is submitted, the flow is executed. The payload of the trigger includes the globalId of the record just created, which is made available in Microsoft Power Automate as dynamic content.

 

The HTTP module is used to encode the URL parameters.

 

 

The HTTP method is set to POST. The URI parameter targets the encodeUrlParams operation in the Survey123 REST API. In the queries group, the token parameter is populated dynamically with the token value provided in the Survey123 webhook payload. The params parameter includes the URL parameters we want encoded. In this case, we pass edit for the mode and the globalId of the feature we want to edit. The globalId is coming as dynamic content from the Survey123 webhook payload as well.

 

Next the Parse JSON module is used, so that later we can easily extract the encoded values from the output of the Survey123 REST API.

 

 

Finally, the Send Email module is used to dynamically create a Survey123 link within the subject of the email. Note that the encoded parameters are dynamically added to the Survey123 link.

 

 

Summary

 

The Survey123 web app supports several URL parameters, allowing you to preload responses to questions, and change the look and feel and behavior of your online surveys. Using the encodeUrlParams operation in the Survey123 REST API, you can encrypt these parameters to avoid manipulation by end users. 

You can generate Survey123 links with encrypted parameters manually right from your browser, programmatically through scripting languages like Python, and dynamically through workflow automation in Microsoft Power Automate and Integromat.

13 Comments
hgaignard
Esri Contributor

Hi Ismael,

Thanks for this capacity.

I tried to encode my URL parameters but I get an error :

Viewing is not possible because the globalId parameter was not provided.

When I request my url with parameters it works : 

https://survey123.arcgis.com/share/************?mode=view&globalId={******************}

With encoding manually, I get the error after second refresh :

https://survey123.arcgis.com/share/************?mode=view&globalId={******************}&encodeUrlParams=true

After first refresh :

https://survey123.arcgis.com/share/************?code=OgR2aWV3

After second refresh : 

Viewing is not possible because the globalId parameter was not provided.

With encoding via Microsoft Power Automate, I get the error in the first opening :

Link in the email :

https://survey123.arcgis.com/share/************?code=OgR2aWV3

Any ideas about this issue ?

Thanks

IsmaelChivite
Esri Notable Contributor

Hi Hugo. I can reproduce this problem too. I will pass a note to the team as a heads up, but please make sure this is logged properly through Esri Technical Support in case other people encounter it. By having the Esri Tech Support BUG number you will also be able to track the resolution status. Thanks a lot!

MatthieuANDRE
New Contributor

Hi Ismael,

Is it possible to do the same thing with Integromat ?

Thanks

hgaignard
Esri Contributor

@MatthieuANDRE, you can use Integromat with HTTP Module (Make a request) + JSON Module (Parse JSON)

hgaignard_0-1606209132257.png

HTTP request details :

hgaignard_1-1606209185478.png

MatthieuANDRE
New Contributor

@hgaignard, thanks for the workflow

How should I configure the Parse JSON module ?

hgaignard
Esri Contributor

@MatthieuANDRE, you need to generate a data structure with the response from the Survey123 REST API. Use this example response in JSON Module > Parse JSON > Add Data structure > Generator:

 

{
 "code": "ShBmaWVsZDpjdXNUcmFja05vchRmaWVsZDpjdXNUcmFja05vPTEyMw==",
 "success": true
}

 

TimLiponis
Esri Contributor

@hgaignard  I am trying to do this as well within integromat using ArcGIS Enterprise with the following parameters:

TimLiponis_0-1606333544759.png

But when I try and run the scenario and parse the response using the JSON as you described above, I am getting the following error:

TimLiponis_1-1606333934190.png

Any thoughts on why this is happening?

Thanks!

hgaignard
Esri Contributor

Hi @TimLiponis

First : Does your HTTP module works ? On your screenshot, I don't see the value of the params field in your HTTP request. It should appear like this :

2020-11-26_09-26-09.png

To make sure your query works, launch your scenario and view the output of your HTTP module. 

2020-11-26_09-38-04.png

2020-11-26_09-39-48.png

Secondly : Check your generated data structure with JSON module. Integromat > Data Structures > Your Data Structure Name > Edit. It should look like this.

2020-11-26_09-29-05.png

Ruth_JiatengXu
Esri Contributor

Hi @hgaignard 

The issue you saw with the error "Viewing is not possible because the globalId parameter was not provided" with encodeUrlParams has been fixed. Could you please help us verify the fix if it is possible?

Thanks,

Ruth

hgaignard
Esri Contributor

@Ruth_JiatengXu, this issue is fixed. 

Thanks !

PatrickHassett
New Contributor

Greetings @IsmaelChivite , @hgaignard , @Ruth_JiatengXu ,

I am working with a portal, trying to follow the python api example notebook. Because I am using portal credentials, I am including 'portalUrl' in my post request:

 

from arcgis.gis import GIS
import json, requests


def encodeParams(token, params, portal_url):
    '''Returns a property code representing encoded url parameters'''
    parameters = {'token': token,
                  'params': json.dumps(params),
                  'portalUrl': portal_url,
                  }
    response = requests.post('https://survey123.arcgis.com/api/encodeUrlParams', data=parameters)
    try:
        return(json.loads(response.text)['code'])
    except: KeyError as e:
        return(response.text)

portal = GIS('home')
my_params = {'field:name': 'Marcus',
             'hide': ['field:phnum'],
             }
encodeParams(portal._con.token, my_params, portal.url)

 

The call to encodeParams returns the following response:

{'error': {'message': 'request to https://<<portalURL>>/sharing/rest/community/self?f=json&token=<<token>> failed, reason: Parse Error: Invalid header value char', 'code': -1, 'details': []}, 'success': False}

But if I enter the request to https://<<portalURL>>/sharing/rest/community/self?f=json&token=<<token>>  into my browser I get a successful response.

 

Any advice or insights would be greatly appreciated. Thanks!

 

IsmaelChivite
Esri Notable Contributor
Ruth_JiatengXu
Esri Contributor

Hi @PatrickHassett 

The error is returned by the encodeUrlParams REST API, and the request is sent to portal to verify the token. Could you please provide more info relates to the portal? 

For example, what is the authentication method used by the portal (IWA/SAML/Built-in)? Can it be accessed from external network?

Thanks,

Ruth