Working with existing Feature Services in ArcGIS Survey123
[Last updated October 11, 2019]
[Last updated March 15, 2021]
Typically, when you publish a new survey with Survey123 Connect or the web designer, a new feature service is created on your behalf. Data submitted with your survey will be stored into that new feature service. We built Survey123 this way because we wanted to simplify as much as possible the process of building new surveys. We did not want people having to worry about feature classes, domains and all of that. Instead, we took an approach where the focus is on adding questions and rules to a form, and leave the creation of the data model to the software.
There are some valid cases however, where you may want to publish a survey that works on top of an existing feature service; In this blog post I will describe how this can be done.
Why would you publish a survey on top of an existing feature service?
There are multiple scenarios where building a survey on top of an existing feature service makes sense. I will describe in more detail some of them:
- Databases: ArcGIS Enterprise allows you to create feature services on top of commercial databases like Microsoft SQL Server, Oracle, etc. There are good reasons why people want data in them although in most cases the key is interoperability. Data in databases can be handled via SQL queries which many software stacks, including ArcGIS, support. In order to have Survey123 add, update or read data from these databases you must first create a feature service on top of them and have your survey work with it.
- Existing data: You may have feature services that were created way before Survey123 existed. By building Survey123 smart forms on top of these feature services, you want to be able not only to add new features, but even update existing ones. A classic example is asset inspections: You may already have feature services with thousands of assets... what if you want to now have Survey123 work against these assets to enable inspections through smart forms?
- One feature service, many surveys: In some scenarios, you may want to create multiple surveys on top of the same feature service. For example, say you have 3 distinct geographic areas where different field crews operate. While you want to consolidate all data in a single feature service, you also want to tailor the survey for each team. Each field crew will use its own flavor of the survey, including a distinct offline basemap, custom defaults and trimmed-down choice lists according to the area at hand.
- Another example is that where you want to use Survey123 to manage the complete life-cycle of an incident: one survey is used to report a case, a second survey is used to review the initial report and may be a third survey is used to close the case. In this scenario, all three surveys work against the same feature service, but each presents or hides different questions according to the purpose of the survey at each step of the cycle.
Pre-requisites
Survey123 can only work against feature services where the supportsApplyEditsWithGlobalID property is set to true. There is a detailed explanation of how to meet this requirement in the https://community.esri.com/groups/survey123/blog/2018/07/17/how-to-turn-supportsapplyeditswithgloabl... blog post, but in short here what you will need:
- Your feature layer must be either hosted in ArcGIS Enterprise or ArcGIS Online.
- If using a multiuser geodatabase, layers can use Branch versioning, but not Traditional versioning. To understand the difference betwen the two, check this help topic.
- Your layer must have an indexed GlobalID field and all relationships, including attachment tables, must leverage GlobalID fields.
How to create a survey on top of an existing feature service?
You can build a survey on top of an existing feature service using Survey123 Connect. First. you will need to be logged-in against ArcGIS Online or your own ArcGIS Enterprise instance. Then simply click on New Survey and select the Feature Service option.
Once you select the Feature Service option, the gallery on the right will show you every feature service shared with your account. For ArcGIS Online, this will include all hosted feature services and in ArcGIS Enterprise, all hosted feature services plus all federated feature services shared with you.
If you want to use a service from an ArcGIS Server instance that is not federated with Portal for ArcGIS, then you will need to follow instructions in this KB article to register your service in your portal through a feature layer item. Otherwise, your service will not be shown in the gallery.
Ok, so next you need to select the feature service of your interest and give your survey a name. Once you click on Create Survey, Survey123 Connect will look into your feature service, checking all fields, related tables and geodatabase domains. A new XLSForm will be created for you as a starting point for your survey. I will call it the initial XLSForm design.
Understanding the initial XLSForm design
In the next few sections I will describe the rules followed by Survey123 Connect to generate the initial XLSForm design. Understanding these rules is important, because you will often want modify the initial XLSForm to tailor the user experience of field users using your survey.
One question per feature layer field:
Other than a few exotic exceptions, that I will describe later, Survey123 Connect will add a new question to your initial XLSForm design for every field in the first layer of your feature service. The questions will be added to the survey XLSForm worksheet in the same order as fields appear in your feature layer. You can reorder the questions in your survey and you can also remove any questions you may want to hide.
Reordering questions can be very handy, because the order of fields in a table may not respond to the logical order in which you want people to enter data in your form. Removing questions in the XLSForm can also be useful, because you may have fields in your feature layer that you do not want to expose to users. Think of internal fields that people in the back-office will use but that are not meant to be visible to field users for example.
XLSForm Question types:
The initial XLSForm design assigns XLSForm question types based on the corresponding feature layer field type, according to the following table:
| esriFieldTypeInteger | integer |
| esriFieldTypeDouble | decimal |
| esriFieldTypeString | text |
| esriFieldTypeDate | datetime |
| esriFieldTypeGeometry (point) | geopoint |
| esriFieldTypeGeometry (polyline) | geotrace |
| esriFieldTypeGeometry (polygon) | geoshape |
| esriFieldTypeGUID | text |
There are some notable exceptions to the rule above.
- First, reserved fields in your feature layer will not be added to your XLSForm: ObjectID, GlobalID and Editor Tracking Fields will be ignored.
- If your layer includes attachments, you will need to add image questions manually.
- Fields in your feature layer that have a coded value domain will be modeled as select_one questions and the corresponding coded values will be automatically added as a list in the choices worksheet.
The correspondence of Esri field types to question types in XLSForm, as described above, is a best guess to ensure data consistency. That is, if you were to publish the survey, you are guaranteed to be able to submit data to all fields in the feature service. However, you may want to adjust the XLSForm types depending on your needs. For example:
- Esri date fields are modeled in XLSForms as datetime, but you can choose to change the XLSForm to just time (if you want to capture time) or date (if you do not care about the time). You should avoid changing the XLSForm question type to decimal or text for example, because the types would not be compatible.
- Esri string fields will be translated by Survey123 Connect as text questions, but you could switch the XLSForm type to integer or decimal for example if you wanted to provide a numeric user input. You could also use question types such as barcode against text fields.
- If attachments are enabled in your feature layer, you can add one or more image or audio type of questions in your XLSForm.
One key concept to remember is that the XLSForm question type fundamentally defines the type of user input that will be presented in the form. As long as the input captures a data type that is compatible with the corresponding target feature layer field, you are fine. For example: Do not expect the calendar control exposed by a date XLSForm question to generate data that is compatible with a field of type decimal. Now, if you feel like you can accelerate data capture using a barcode question against a string field, you can replace the initial text type of question with a barcode question because barcode question types will always return an output that can be stored as a string. You can also use the appearance XLSForm column to refine the user input experience.
XLSForm bind::esri:fieldType and fieldLength:
If you look carefully at your XLSForm, you will also note that Survey123 Connect added in the bind::esri:fieldType column the exact Esri data type found for fields in your feature layer. The bind::esri:fieldLength column will also contain the length originally set to your feature layer fields.
You should never change the esri:fieldType, because they are defined by the feature layer and Connect will not be able to change your layer.
You could change the length, as long you make it shorter. Making the length shorter in XLSForm allows you to limit the user input. For example, if your field has a length in the feature layer of 200 characters but you want to limit the user input from your survey to 5, simply change the bind:esri:fieldLength value to 5. You are not allowed to specify a fieldLength in your XLSForm that exceeds the actual length of the field in your feature layer.
You will rarely have to change the bind::esri:fieldLength and bind::esri:fieldType columns. If you do, you may trigger an error when attempting to publish the survey. The reason is that Survey123 Connect has a built-in check to ensure that a published survey is compatible with the underlying schema of your feature service. If the schema are not compatible, then you will not be allowed to publish the survey. Tinkering with the bind::esri:fieldLength and in particular with bind::esri:fieldType can easily result on schema inconsistencies.
Question Names and Labels:
The XLSForm name column reflects the exact field names found in your feature layer. You cannot change the values in this column. The label column is populated using the field name alias from your feature layer. You can do whatever you want with the label column (except leaving it empty). You may want to rephrase questions, embed html formatting etc, etc
Repeats (related tables):
In the event that the first layer in your feature service is related to other layers or standalone tables, you will see the presence of repeats in the XLSForm. In Survey123 we model related geodatabase tables/layers as repeats. The name of the repeat will be defined by the table name of the related table or layer and cannot be changed. The label of the repeat can contain any text you like.
Within the XLSForm repeat block, you will find questions corresponding to the fields found in the related child table. The rules for how questions are added within a repeat are exactly the same as for questions in the main feature layer. The rules for changing question types, labels etc as well as for removing or reordering them are the same too. Just be aware you cannot take questions outside of the repeat (removing them is fair game, but not to take them out of the repeat block and into the main block of the form)
Select_ones:
I want to add a few notes on how select_one questions behave. In short, every time a field in your feature service uses a coded value domain, a new select_one type of question is created. The coded value domains are matched by a corresponding XLSForm list in the choices worksheet. You are free to reduce the number of choices in the list if that makes sense for you. You can also change the labels in the list as you see fit.
The values in the name column of the choices worksheet need to remain unchanged because otherwise, you will be adding data into your feature service outside the valid values set by your geodatabase domain.
Range domains are modeled as range questions in XLSForms.
Geodatabase subtypes and contingent values are ignored by Survey123, although you can use cascading selects to model them.
Other XLSForm columns and question types:
As far as the survey worksheet in your XLSForm, columns and question types other than the ones described above will not be used. That is, the default, calculation, constraint, relevant columns etc will be empty. Question types such as groups, notes or calculates will also never be present in the initial XLSForm design. Now: you can, and should, use those extra columns and question types as appropriate to build a great data capture experience. For example:
- You can use XLSForm groups and pages to break down questions in your survey into logical sections. This simplifies and accelerates data capture without altering the original geodatabase schema in your feature layer.
- You can leverage XLSForm notes and hints to provide field users with critical information to capture data more efficiently. Similarly, notes and hints have no effect on the geodatabase schema.
- You can use constraints, relevant statements as well as input masks, calculations and defaults to better control user input.
- Appearances are also a great way to optimize the field data capture experience.
The initial XLSForm design is just that: a starting point. Take pride of your surveys and transform flat GIS data models into beautiful smart forms.
Once you are happy with your design, you will proceed to publish your survey. When publishing, no new feature service will be created for your form. Instead, the feature service you selected will be referenced by your survey.
Settings worksheet:
The settings worksheet of the initial XLSForm contains two key pieces of information:
- submission_url: This column defines the url of the feature layer item in your ArcGIS Organization on which responses to your survey will be stored. This is the exact url of the feature layer you chose in the Create New Survey dialog as the seed of your survey. Note that this is the url to the feature layer item, rather than the url of the feature service itself.
- form_id: This column defines the name of the layer in your feature service that your survey will hit. If empty, your initial XLSForm design will always point to the first layer found in your feature service.
While you often may not want to change the submission_url, there is one use case where changing the form_id is very handy. Think for example, that you want to create a survey on top of a related standalone table. For clarity, lets think of a feature service with parcels (polygons) and their related assessments. You may not want to build a survey to edit the parcels, but you may want to create one to create the assessments. Typically, when you get the initial XLSForm design, you will have questions in your survey for all the fields in the parcels layer and then a repeat group, with all questions in the related assessment table. This is only because in a feature service, typically the layer with geometries (parcels in our case) goes first, and then the related tables. To create your parcel assessment survey you will delete all questions corresponding to the parcels layer from your survey and then take the name of your assessments repeat and put it into the form_id setting. Finally you will delete the begin_repeat and end_repeat rows in your survey.
To ensure that assessment data is properly attached to the corresponding parcel, it is important that in your survey you preserve a question to enter the unique identifier of the parcel. You will need to know what field in the child assessment table is used for the geodatabase relationship.
Limitations when working with Repeats in your XLSForm:
Mapping the ArcGIS geodatabase model into an XLSForm is possible to some extent, but there are some limitations to bear in mind. In the case that you are working with existing feature services published from ArcGIS Desktop, there are chances that you could run into some situations where you simply cannot build a survey on top of a feature service, or at least not the exact survey that you had in mind. The truth is that you can actually build surveys pretty much on top of any feature service, except when working with related tables. Lets get into the details:
- The XLSForm specification does not accept two questions in a single survey with the same name. This makes sense because otherwise you could not uniquely reference the question in your XLSForm expression. When working with existing feature services this will be a problem when you have related tables AND fields in both tables with the same name. In some cases this is not a real problem because you may not want to include the repeat on the survey. You may also be able to delete one of the questions referencing the duplicated field name (from either the main body of the survey, or from within the repeat). But this is not always the case...
- The XLSForm specification does not accept spaces in the name of a repeat. This is a problem when you have a related table in your feature layer that includes spaces in the name. If you encounter this you can do two things: One is obviously to delete the repeat group from your survey all together. If you cannot delete the repeat then you can ask the owner of the feature service to publish the feature service again, eliminating the space in the name of the related table. Of course this may not be possible in many cases, but I bring this up because changing the name of a layer in the feature layer does not imply changing the name of the table in the database. All you need to do is to change the name of the layer in the table of contents in ArcGIS Desktop and publish again.
- The XLSForm specification does not like spaces in the name column of the choices worksheet: This will be a problem when you have coded values in a domain that include spaces in them. Fortunately, you will only get a warning when validating your XLSForm in Connect. You will actually be able to publish your survey with no problem, but it is not good practice to use spaces. It is also technically possible to encounter coded values in domains that exceed the maximum length of 32 characters allowed by choice names.
Some tips and other things to be aware of:
- In the old days of Survey123, people used to create the initial XLSForm designs by hand and from scratch. If you really understand the meaning of the submission_url and form_id XLSFom settings and you understand how to map fields in your feature service to XLSForm question types, you can do it. Typically using the Create Survey from Feature service option as described above is the best way to go, but forcing yourself to build a survey from scratch on top of an existing feature service will be a good learning experience.
- Sharing can be problematic! Keep in mind that while you will always be the owner of the survey you are publishing, you may not be the owner of the underlying feature service. Unless you are the owner, you can't really control the sharing, so you will want to make sure that your feature service is made available to users that will access your survey.
- Capabilities: Similarly, unless you are the owner of the feature service, you will have no control over the update, query, delete and add capabilities of the feature service. It is your responsibility to make sure your survey is consistent with the capabilities enabled in the service. For example, if the feature service does not support updates, then you should not enable the Inbox.
- Hosted Feature Layer Views: You can create surveys on top of hosted feature layer views. Views are a clever way to securely limit access to features and or columns in an existing feature service. Views also allow you to better control sharing of your feature service. There are some scenarios where you may want to create surveys on top of views, rather than directly on top of the feature service.