Securing data in public surveys (Survey123 Connect)

5334
14
05-11-2020 08:23 AM
IsmaelChivite
Esri Frequent Contributor
5 14 5,334

By definition, a public survey is accessible to anyone who wants to submit data to it, but that does not mean that anyone should also be able to look at the data itself. If your public Survey123 form contains sensitive information, you should configure your survey to prevent users in the public domain from downloading, querying or changing already submitted data. Unfortunately, it is not uncommon to find public surveys where the security configuration of the survey is not set appropriately, allowing unauthorized access to the survey’s data. 

This article describes best practice for securing the data of public surveys published from Survey123 Connect. If you are interested in securing data for a public survey published with the Survey123 web designer, refer to https://community.esri.com/groups/survey123/blog/2020/05/11/securing-data-in-public-surveys-survey12... 

If you are not familiar with the basics of public surveys, refer to https://community.esri.com/groups/survey123/blog/2016/11/10/getting-started-with-public-surveys.

A bit of context before we start

To properly secure your survey results it is important to understand first some basic concepts.  When you publish a survey using Survey123 Connect, a new folder is created in your ArcGIS account. This folder includes the name of your survey so you can easily find it.  Inside this folder, you will find a Form item and a Feature layer item:

  • Form item: The Form item contains the definition of the questionnaire presented to users: The labels of your questions, the calculations, media files and other resources needed to render your form.
  • Feature layer: The feature layer is the item where responses to your survey are stored.

In short, the survey folder contains one item (the form item) for the survey questions and one item (the feature layer) for the survey responses.

If you are working with sensitive data, you never want to share your surveys source feature layer. Instead, you will want to keep your survey feature layer private, and build feature layer views on top where you can better control the sharing and privilege properties. At the very least, you will want to create two feature layer views:

  • A view for the Survey123 web and field apps to use.  This view will allow the apps to add, and if appropriate to edit records in your feature layer.
  • A view for the Survey123 website to use. This view will control who can access the survey results through the Survey123 website, and with what privileges: just view, or also view and edit.

Additionally, you may want to create extra views to support other applications, such as ArcGIS Dashboards, Web AppBuilder apps, etc.

This article describes in detail how to build these feature layer views and associate them with your survey. If you are not familiar with the concept of feature layer views, I suggest read the Create hosted feature layer views—ArcGIS Online Help | Documentation help topic.

Do not create the views too early.

Let me put this upfront: as of version 3.9, Survey123 Connect does not like views: Connect does not create views, and does not handle them well when you delete or modify the survey. This is something that is going to change, rendering this whole article unnecessary, but for now bear with me.

As stated above, my recommendation is that you always use feature layer views when your survey is shared with people, but from a practical perspective you do not want to create the views too early. Survey design is an iterative process where you will be adding, changing and removing questions from your survey frequently. Some of these changes necessarily affect the schema of the surveys feature layer. If your survey is configured with views, Connect may not be able to change the schema of the source layer. If the schema of the layer is changed, your views will break.

For this reason, keep your survey without views for as long as you are working on it and configure the views when you are ready to put your survey in production, right before you share your survey with users.

Create a view for the Survey123 website first

I said before you will want to create at least two views: One for users who will look at your survey results through the Survey123 website, and another one for users to submit data through the Survey123 web and/or field apps. It is best to start with the view to control access to the survey results.

To build the view:

  • Login into the Survey123 website
  • Navigate to the Collaborate tab of your survey
  • Switch to the Viewer panel and hit Save.

If you return now to the Survey123 folder under Content in ArcGIS.com, you will notice that a new view has been created for your survey. This view has a 'stakeholder' suffix.

This new view will control who can use the Survey123 website to look at your survey results.  You will want to use the Viewer panel in the Survey123 website to control this. For example, say that users in a group called 'City of Cilantro' need to be able to look at the results of your survey, create reports and download data. Then you will go into the Collaborate tab, switch to the Viewer panel and share your survey results with that group. At that point, 'City of Cilantro' users can log into the Survey123 website and use the Overview, Data and Analyze tabs to do what they need. 

I would not recommend that you modify the sharing of this view through ArcGIS.com. For the Survey123 website to properly work, the sharing of the Form and stakeholder view items must be in sync.  The Collaborate tab in the Survey123 website takes care of that.

Create a view for the Survey123 web and field apps next

Configuring the view for the Survey123 web and field apps is a bit more involved. We need to create this view manually, then associate the view with the Survey123 Connect survey.

  • Log into the arcgis.com website and click on the My Content tab.
  • Click on the Form item to open its item details page.
  • Look for the Layers section and click on your survey feature layer link. This will open the item details page for your surveys feature layer.
  • Click on Create View Layer. You can choose any title for your feature layer view.

  • Go into the Settings tab and enable Editing. Optionally, you can also choose what type of editing is allowed. At the very least, Survey123 will need adding permissions.

To make your survey work against your own  feature layer view, you need to configure the submission_url and form_id XLSForm settings in your survey. This can be an error prone process at first. Once you are familiar with this I am sure you will do this with your eyes closed, but here I am going to follow a long but safe route:

  • In Survey123 Connect, from the survey gallery, click on New Survey and then choose the Feature Service option.
  • Look for the feature layer view you just created and give your new survey a throw-away name, such as temp or delete_me.

  • Open the XLSForm of your temporary new survey and switch to the settings worksheet. Then copy the values in the form_Id and submission_url cells into a text editor or a safe place, so we can paste them later into the original survey.

The submission_url value defines the feature layer (or feature layer view) that the survey is targeting. If empty, Survey123 Connect will create a new feature layer when you publish the survey. If a value is provided, the survey is published targeting the specified layer by the submission_url . The form_id value defines the sub-layer in your feature layer that drives the questions in your survey.

  • Back in Survey123 Connect go back to survey gallery, and open the survey that you want to make public.
  • Open the XLSForm, paste the submission_url and form_id values.
  • Save your XLSForm and publish your survey again.

The Publish dialog will indicate that your existing survey will be updated to use a custom feature service as specified by the submission URL, as shown in the next screenshot.

Now that your survey has been updated to target your own feature layer view, you can share your survey publicly with confidence. We will do that from the Survey123 website.

Sharing your survey publicly

  • Log into the Survey123 website at survey123.arcgis.com.
  • From the survey gallery, open the Collaborate tab of your survey

  • The Submitter panel controls who can submit data to your survey. While in the Submitter panel, look for the section named 'Who can submit to this survey?' and check the Everyone (Public) option to share your survey publicly. 

If the option to share your survey publicly is missing, contact your ArcGIS administrator.

  • Scroll down the page and look for the 'What can submitters do?' section. Check if not already the 'Only add new records' option and click on Save at the bottom to persist all changes.

At this moment, your survey is shared publicly, allowing anyone to submit data through both the Survey123 web and field apps. You can get the link to your survey from the top of the Collaborate tab and distribute the link with your users. Since you have restricted access to 'Only add new records' in the Collaborate tab, it will not be possible to query, update, delete or download your survey data through the Survey123 web or field apps. Your survey's feature layer will also be secure, preventing any type of access (other than adding new records) from other Esri apps, third party apps or programmatic access.

If you go back to My Content in arcgis.com and check your survey folder, you will find that your feature layer view and the Form item are now shared publicly, while the feature layer remains shared only with you, the owner. This is the way you want it. Do not share the source feature layer if you want to keep your data safe.

Sharing your survey results in web applications and dashboards

The same technique we used to create a feature layer view for the Survey123 web and field apps can be replicated to support other apps and uses. It is not good practice to reuse the feature layer view we just created or to share the source feature layer. Build new views, restrict access to data as appropriate to the needs of the web app and share accordingly.

Here are a few links to learn more about feature layer views:

Tags (2)
14 Comments
by Anonymous User
Not applicable

Thanks for the step-by-step! I'm definitely keeping this in my back pocket! Question though: what about public surveys (with views) that are already published (with the radio button "always use the latest version" marked)? I've created a survey in Connect 3.7 that can only display correctly if that radio button is marked, and have used views to keep data private. I've tested these views/layers for privacy concerns and they worked in 3.8.

ColinCampbell
Occasional Contributor

Thanks for this Ismael Chivite‌.   I have two questions:

Firstly, you say 'Connect does not create views, and does not handle them well when you delete or modify the survey. This is something that is going to change,'.  Is there an rough estimate when this might be the case?

Secondly, in the past when doing public surveys we've been guided to use the feature layer directly (i.e. not creating a view) and to alter the settings on that to prevent public access to the data (using a settings setup like shown in the attached image).  Whilst we'll use views in the future, has our data been insecure if we have shared the feature service publicly but using the settings as shown below?  

IsmaelChivite
Esri Frequent Contributor

Hi. Generally speaking, I would recommend that you publish your survey again when you want it to be on the latest version. The issue with the 'always use the latest version' option is that 1) the upgrade process happens dynamically at runtime, potentially slowing down the initial load of the form and 2) you always run into the risk of having new versions alter the behavior of your form.

You can use the version query parameter to try your form in different versions: ?version=3.8 for example. If you feel comfortable, simply publish again and the webform will be upgraded to the latest release. Of course, make sure you also have the latest copy of Survey123 Connect installed.

The version of the web app used to render your form should not have any play into how feature layer views are used.

IsmaelChivite
Esri Frequent Contributor

Two great questions:

  • When will Connect be tolerant to views? My statement above was to acknowledge that we do not feel comfortable with the way Connect behaves today. I do not have an exact timeframe for when this will be addressed, but it is present in our planning. At this moment we are focused on 3.10 for release in July and there is no room for us to incorporate this into the release. It will be after 3.10 for sure. We will probably have more details about this later in the year.
  • In the past I was told to set permissions in the feature layer and now you tell me to set permissions in a view. Can you describe why?  You can certainly set permissions in your feature layer as you describe and not use a view. That will work and be secure, but only as long as you never change these permissions in the feature layer.  You as the owner of the survey will always be able to query, delete and update features in the survey but nobody else will. Your data is really locked down to your account right now. It is secure, but so locked down only you as the owner can see it.  If in the future someone asks you to create a web application to display the results, you will be forced to change these privileges and at that point, anyone will be able to query your data. That puts your data at risk, at that moment.   Using views and unsharing your source feature layer gives you more flexibility to tailor the privileges granted to each group of people needing access to your survey and or data.

Hope it helps.

JeremyJohnson5
New Contributor III

Thank you for Ismael Chivite for sharing this process. This is definitely the process that i will be using now.  Previously i would change the settings of the feature layer but by taking the time to follow the steps and create views will allow me to share the data more confidently with out the worry of the data being altered inadvertently. 

Todd
by
New Contributor III

Great write up Ismael Chivite‌. Interested if you will be updating this post, particularly the Create a view for the Survey123 web and field apps next section?

After creating my survey and related objects I decided to do a complete forklift/overhaul, for security sake, and follow these directions as it relates to the views. However, this step: Click on the Form item to open its item details page, then click on Create View Layer. You can choose any title for your feature layer view. does not track. If you click on the form item there is no 'Create View Layer'. Instead you'd have to click on the Feature Layer.

Thanks!

JamesTedrick
Esri Esteemed Contributor

Hi Todd,

There is an intermediate step - Click on the form item, then click on the entry in the 'Layers' section.  That will take you to the feature layer from which views can be created (if published by Survey123 Connect).

Todd
by
New Contributor III

Thanks James, figured it out, but it could cause confusion for someone who may not be familiar with the software. Appreciate the reply.

Todd
by
New Contributor III

Hey James, are you able to get this process to work with 3.9? I decided to revisit this and create a project from scratch but it will not allow data submissions.

Note: In survey123.arcgis.com I'm using the 'Open the survey in the Survey123 field app directly.' option.

This is what my ArcGIS Online Content folder contains after following the above post:

  • Form - public shared
  • Feature Layer (hosted, view) - public shared
  • Feature Layer_stakeholder (hosted, view) - owner shared
  • Web Map - owner shared
  • Feature Layer (hosted) - owner shared

Scratching my head. Thanks for any input!

Todd

Todd
by
New Contributor III

UPDATE: In ArcGIS Survey123 online under 'Share Survey' - when I changed the 'What can submitters do?' option from 'Only add new records' to 'Add and update records' it works fine. However, as Ismael Chivite mentioned you cannot edit the schema at this point unless you want to start over.

BritaVespere
New Contributor III

Does the Collaboration settings (for example, to specify survey open/close dates) work for the latest Survey123 Connect version 3.11. created surveys? I get the error "Cannot read property 'success' of undefined"...

Also:

I understand that if I desire to add a Thank You screen info, theme - I have to do it for a survey before I download it and create its content in Connect? 

The images for background, for Thank You screen are lost after download to Connect - if uploaded. From link works fine.

When I download Web Designer created survey to Connect the Excel file is still the old version (lighter green look, no added tabs etc.)

Would the workflow for public, securely used in browser surveys be changed in the near future? 

Otherwise, really enjoy the S123 functionality!

chaims
by
New Contributor II

I'm still not sure I understand the need for creating the first view - the one that is intended for the web/mobile app to use, and requires updating the submission url: As pointed out, by changing the permissions on the original feature layer (either directly through arcgis.com, or through the survey123 website), the same functionality can be secured. If at a later point the need arises to share the data with another user, or through another app, I would then create the view, at that point, and make any required updates on that view. In a previous response, it was mentioned that this would compromise the security of the source feature layer, but I can't see how that would be the case? 

SanFranciscoBayConservationDev
New Contributor

Hi, thanks for helping address this need. Curious if the workflow/guidance has changed at all in recent versions. When I go to the Collaborate tab in website from a survey published in Connect 3.9, I don't see the Submitter/Viewer tabs. Instead, Share Survey/Share Results are available options. Is this where you control the permissions now?

MarcusVDSilva
New Contributor II

Hi @IsmaelChivite !

Is this workflow is still valid at version 3.12?

Thanks in advance!