Securing data in public surveys (Survey123 Connect)

IsmaelChivite · ‎05-11-2020

[Updated October 27, 2021. Screenshots updated to reflect latest state of Connect and website]

By definition, a public survey is accessible to anyone who wants to submit data to it, but that does not mean you need to make your survey results public as well. In fact, the default configuration of a public survey always keeps your survey results private. This ensures that users in the public domain cannot view, query, download or update already submitted data.

Unfortunately, it is not uncommon to find public surveys where the security configuration of the survey is not set appropriately, allowing unauthorized access to the survey’s data. This article describes best practice for securing the data of surveys published with Survey123 Connect.

If you are interested in securing data for a public survey published with the Survey123 web designer, refer to https://community.esri.com/groups/survey123/blog/2020/05/11/securing-data-in-public-surveys-survey12...

If you are not familiar with the basics of public surveys, refer to https://community.esri.com/groups/survey123/blog/2016/11/10/getting-started-with-public-surveys.

A bit of context before we start

To properly secure your survey results it is important to understand first some basic concepts. When you publish a survey using Survey123 Connect, a new folder is created in your ArcGIS account. This folder includes the name of your survey so you can easily find it. Inside this folder, you will find a Form item and a Feature layer item:

Form item: The Form item contains the definition of the questionnaire presented to users: The labels of your questions, the calculations, media files and other resources needed to render your form.
Feature layer: The feature layer is the item where responses to your survey are stored.

In short, the survey folder contains one item (the form item) for the survey questions and one item (the feature layer) for the survey responses.

If you are working with sensitive data, you never want to share your surveys source feature layer. Instead, you will want to keep your survey feature layer private, and build feature layer views on top where you can better control the sharing and privilege properties. At the very least, you will want to create two feature layer views:

A view for the Survey123 web and field apps to use. This view will allow the apps to add, and if appropriate to edit records in your feature layer.
A view for the Survey123 website to use. This view will control who can access the survey results through the Survey123 website, and with what privileges: just view, or also view and edit.

Additionally, you may want to create extra views to support other applications, such as ArcGIS Dashboards, Web AppBuilder apps, etc.

This article describes in detail how to build these feature layer views and associate them with your survey. If you are not familiar with the concept of feature layer views, I suggest read the Create hosted feature layer views—ArcGIS Online Help | Documentation help topic.

Do not create the views too early.

As stated above, my recommendation is that you always use feature layer views when your survey is shared with people, but from a practical perspective you do not want to create the views too early. Survey design is an iterative process where you will be adding, changing and removing questions from your survey frequently. Some of these changes affect the schema of the surveys feature layer. If your survey is configured with views, Connect will not be able to change the schema of the source layer.

For this reason, I suggest you keep your survey without views for as long as you are working on is design. Configure the views when you are ready to put your survey in production, right before you share your survey with users.

Create a view for the Survey123 website first

I said before you will want to create at least two views: One for users who will look at your survey results through the Survey123 website, and another one for users to submit data through the Survey123 web and/or field apps. It is best to start with the view to control access to the survey results.

To build the view:

Login into the Survey123 website
Navigate to the Collaborate tab of your survey
Switch to the Share results panel, select who should have access to your survey results and what privileges you want to grant to these users. Click Save.

If you return now to the Survey123 folder under Content in ArcGIS.com, and refresh it, you will notice that a new view has been created for your survey. This view has a 'stakeholder' suffix.

This new view will control who can use the Survey123 website to look at your survey results. You will want to use the Share results panel in the Collaborate tab of the Survey123 website to control this. For example, say that users in a group called 'City of Cilantro' need to be able to look at the results of your survey, create reports and download data. Then you will go into the Collaborate tab, switch to the Share results panel and share your survey results with that group. At that point, 'City of Cilantro' users can log into the Survey123 website and use the Overview, Data and Analyze tabs to do what they need.

I would not recommend that you modify the sharing of this newly created view through ArcGIS.com. For the Survey123 website to properly work, the sharing of the Form and stakeholder view items must be in sync. The Collaborate tab in the Survey123 website takes care of that.

Create a view for the Survey123 web and field apps next

Configuring the view for the Survey123 web and field apps is a bit more involved. We need to create this view manually, then associate the view with the Survey123 Connect survey.

Log into the arcgis.com website and click on the My Content tab.
Click on the Form item to open its item details page.
Look for the Layers section and click on your survey feature layer link. This will open the item details page for your surveys feature layer.
Click on Create View Layer. You can choose any title for your feature layer view.

Go into the Settings tab
Enable Public Data Collection and save.
Make sure Enable editing is checked and save.

Now that your feature layer has been created, you will need to use it in your survey.

Configuring your survey to use the view you just created

To make your survey work against your own feature layer view, you need to configure the submission_url and form_id XLSForm settings in your survey. This can be an error prone process at first. Once you are familiar with this I am sure you will do this with your eyes closed, but here I am going to follow a long but safe route:

In Survey123 Connect, from the survey gallery, click on New Survey and then choose the Feature Service option.
Look for the feature layer view you just created and give your new survey a throw-away name, such as temp or delete_me.

Open the XLSForm of your temporary new survey and switch to the settings worksheet. Then copy the values in the form_Id and submission_url cells into a text editor or a safe place, so we can paste them later into the original survey.

The submission_url value defines the feature layer (or feature layer view) that the survey is targeting. If empty, Survey123 Connect will create a new feature layer when you publish the survey. If a value is provided, the survey is published targeting the specified layer by the submission_url . The form_id value defines the sub-layer in your feature layer that drives the questions in your survey.

Back in Survey123 Connect go back to survey gallery, and open the survey that you want to make public.
Open the XLSForm, paste the submission_url and form_id values.
Save your XLSForm and publish your survey again.

The Publish dialog will indicate that your existing survey will be updated to use a custom feature service as specified by the submission URL, as shown in the next screenshot.

Now that your survey has been updated to target your own feature layer view, you can share your survey publicly with confidence. We will do that from the Survey123 website.

Sharing your survey publicly

Log into the Survey123 website at survey123.arcgis.com.
From the survey gallery, open the Collaborate tab of your survey

The Share survey panel controls who can submit data to your survey. While in the Share survey panel, look for the section named 'Who can submit to this survey?' and check the Everyone (Public) option to share your survey publicly.

Scroll down the page and look for the What can submitters do? section. Note that by default, the selected option is 'Only add new records'. This guarantees that your survey layer will not expose your survey results.
Click on Save at the bottom to persist all changes.

At this moment, your survey is shared publicly, allowing anyone to submit data through both the Survey123 web and field apps. You can get the link to your survey from the top of the Collaborate tab and distribute the link with your users. Since you have restricted access to 'Only add new records' in the Collaborate tab, it will not be possible to query, update, delete or download your survey data through the Survey123 web or field apps. Your survey's feature layer will also be secure, preventing any type of access (other than adding new records) from other Esri apps, third party apps or programmatic access.

If you go back to My Content in arcgis.com and check your survey folder, you will find that your feature layer view and the Form item are now shared publicly, while the feature layer remains shared only with you, the owner. This is the way you want it. Do not share the source feature layer if you want to keep your data safe.

Sharing your survey results in web applications and dashboards

The same technique we used to create a feature layer view for the Survey123 web and field apps can be replicated to support other apps and uses. It is not good practice to reuse the feature layer view we just created or to share the source feature layer. Build new views, restrict access to data as appropriate to the needs of the web app and share accordingly.

Here are a few links to learn more about feature layer views: