ichivite-esristaff

Working with existing Feature Services in Survey123 for ArcGIS

Blog Post created by ichivite-esristaff Employee on Sep 25, 2017

Typically, when you publish a new survey with Survey123 Connect or Web Designer, a new feature service is created on your behalf. Data submitted with your survey will be stored into that new feature service. We built Survey123 this way because we wanted to simplify as much as possible the process of building new surveys. We did not want people having to worry about feature classes, domains and all of that. Instead, we wanted to take a top-down approach where the focus is on building a great smart form, and leave the creation of the data model to the software.

 

There are some valid cases however, when you want to publish a survey that works on top of an existing feature service; In this blog post I will describe how this can be done. 

 

Why would you ever publish a survey on top of an existing feature service?

 

There are multiple scenarios where building a survey on top of  an existing feature service  makes sense. I will describe in more detail some of them:

 

  • Databases: ArcGIS Enterprise allows you to create feature services on top of commercial databases like Microsoft SQL Server, Oracle, as  well other more exotic ones such as Altibase, DB2, PostgreSQL, Netezza, Informix, Teradata, Dameng, SAP Hana and Informix. There are good reasons why people want data in them although in most cases the fundamental is interoperability. Data in databases can be handled via SQL queries which many software stacks, including ArcGIS, support. In order to have Survey123 add, update or read data from these databases you must first create a feature service on top of them and have your survey work with it.
  • Existing data: You may have feature services that were created way before Survey123 even existed.  By building Survey123 smart forms on top of these feature services, you want to be able not only to add new features, but even update existing ones. A classic example is asset inspections: You may already have feature services with thousands of assets... what if you want to now have Survey123 work against these assets?
  • One feature service, many surveys: In some scenarios, you may want to create multiple surveys on top of the same feature service.  For example, say you have 3 distinct geographic areas where people are  capturing data and you have assigned one team to each area.  While you want to consolidate all data in a single feature service, you also want to tailor the survey for each team.   Each field  crew will use its own flavor of the survey, including a distinct offline basemap, custom defaults and trimmed-down choice lists according to the area at hand.  Another classic example is where you want to use Survey123 to manage the complete life-cycle of an incident, where one survey is used to report a case, a second survey is used to review the initial report and may be a third survey is used to close the case.  In this scenario, all three surveys work against the same feature service, but each presents or hides different questions according to the purpose of the survey at each step of the cycle.

 

How to create a survey on top of an existing feature service?

 

You can build a survey on top of an existing feature service using Survey123 Connect for ArcGIS.  First. you will need to be logged-in against ArcGIS Online or your own ArcGIS Enterprise instance.  Then simply click on New Survey and select the Feature Service option.

 

 

Once you select the Feature Service option, the gallery on the right will show you every feature service shared with your account.  For ArcGIS Online this will include all hosted feature services and in the case of ArcGIS Enterprise, all hosted feature services plus all federated feature services shared with you. 

If you want to use a service from an ArcGIS Server instance that is not federated with Portal for ArcGIS, then you will need to follow instructions in this KB article to register your service in your portal through a feature layer item.  Otherwise, your service will not be shown in the gallery.

                

Ok, so next you need to select the feature service of your interest and give your survey a name.  Once you click on Create Survey, Survey123 Connect will look into your feature service, checking all fields, related tables and geodatabase domains. A new XLSForm will be created for you as a starting point for your survey. I will call it the initial XLSForm design.

 

Understanding the initial XLSForm design

 

In the next few sections I will describe in details the rules followed by Survey123 Connect to generate the initial XLSForm design. Understanding these rules is important, because you will often want modify the initial XLSForm design to tailor the user experience of field users using your survey.

 

One question per feature layer field:

 

Other than a few exotic exceptions, that I will describe later, Survey123 Connect will add a new question to your initial XLSForm design for every field in the first  layer of your feature service.  The questions will be added to the survey XLSForm worksheet in the same  order  as fields in your feature layer.  You can reorder the questions in your survey at will and you can also remove any questions you may want to hide.  Reordering questions can be very handy, because the order of fields in a table may not respond at all to the logical order in which you want people to enter data into your form.  Removing questions can also be useful, because you may have fields in your feature layer that you may not want to expose in the survey at all. Think of internal fields that people in the back-office will use but that are not meant to be visible to field users.

 

XLSForm Question types:

 

The initial XLSForm design assigns the XLSForm question types based on the corresponding feature layer field type, according to the following table:

 

Feature layer field typeQuestion type in initial XLSForm design
esriFieldTypeIntegerinteger
esriFieldTypeDoubledecimal
esriFieldTypeStringtext
esriFieldTypeDatedatetime
esriFieldTypeGeometry (point)geopoint
esriFieldTypeGUIDtext

 

There are some notable exceptions to the rule above.

 

  • First, reserved fields in your feature layer will not be added to your XLSForm: ObjectID, GlobalID and Editor Tracking Fields will be ignored.
  • If your layer includes attachments, you will need to add image questions manually.
  • Fields in your feature layer that have a coded value domain will be modeled as select_one questions and the corresponding coded values will be automatically added as a list in the choices worksheet.

 

The correspondence of Esri field types to question types in XLSForm, as described above, is a best guess to ensure data consistency. That is, if you were to publish the survey, you are guaranteed to be able to submit data to all fields in the feature service.  However, you may want to adjust the XLSForm types depending on your needs. For example:

 

  • Esri date fields are modeled in XLSForms as datetime, but you can choose to change the XLSForm to just time (if you want to capture time) or date (if you do not care about the time).  You should avoid changing the XLSForm question type to decimal or text for example, because the types would not be compatible (kind of difficult to reliably store a user entered string into a date type of field).
  • Esri string fields will be translated by Survey123 Connect as text fields, but you could actually switch the XLSForm type to integer or decimal for example if you wanted to provide a numeric user input.  You could also use question types such as barcode against text fields.
  • If attachments are enabled in your feature layer, you can add one or more image or audio type of questions in your XLSForm.

 

One key concept to remember is that the XLSForm question type fundamentally defines the type of user input that will be presented in the form. As long as the input captures a data type that is compatible with the corresponding target feature layer field, you are fine.  For example: Do not expect the calendar control exposed by a date XLSForm question to generate data that is compatible with a field of type decimal.  Now, if you feel like you can accelerate data capture using a barcode question against a string field, you can replace the initial text type of question with a barcode question because barcode scanning will always return an output that can be stored as a string.

 

XLSForm bind::esri:fieldType and fieldLength:

 

If you look carefully at your XLSForm, you will also note that Survey123 Connect added in the bind::esri:fieldType column the exact Esri data type found for fields in your feature layer.  The bind::esri:fieldLength column will also contain the lengths originally set to your feature layer fields.

 

You should never change the esri:fieldType, because they are defined by the feature layer and Connect will not be able to change your layer (and not that you want to).

 

You could change the length, as long you make it shorter. Making the length shorter in XLSForm allows you to limit the user input. For example, if your field has a length in the feature layer of 200 characters but you want to limit the user input from your survey to 5, simply change the bind:esri:fieldLength value to 5.  You are not allowed to specify a fieldLength in your XLSForm that exceed the actual length of the field in your feature layer.

 

You will rarely have to change the bind::esri:fieldLength and bind::esri:fieldType columns. If you do, you may trigger an error when attempting to publish the survey.  The reason is that Survey123 Connect as a built-in check to ensure that a published survey is compatible with the underlying schema of your feature service. If the schema are not compatible, then you will not be allowed to publish the survey.  Tinkering with the bind::esri:fieldLength and in particular with bind::esri:fieldType can easily result on schema inconsistencies.

 

Question Names and Labels: The XLSForm name column reflects the exact field names found in your feature layer. You cannot change the values in this column.  The label column is populated using the field name alias from your feature layer. You can truly do whatever you want with the label column (except leaving it empty).  You may want to rephrase questions, embed  html formatting etc, etc   

 

Repeats:

 

In the event that the first layer in your feature service is related to other layers or standalone tables, you will see the presence of repeats in the XLSForm.  In Survey123 we  model related geodatabase relationships as repeats.  The name of the repeat will be defined by the table name of the related table or layer and cannot be changed. The label of the repeat can contain any text you like.

 

Survey123 only supports one level of repeat. That is, if you have two nested related tables, Survey123 will not be able to model that. Nested repeats are planned to be supported in version 3.0 (July 2018)

              

 

Within the XLSForm repeat block, you will find questions corresponding to the fields found in the related child table. The rules for how questions are added within a repeat are exactly the same as for questions in the main feature layer. The rules for changing question types, labels etc as well as for removing or reordering them are the same too.  Just be aware you cannot take questions outside of the repeat (removing them is fair game, but not to take them out of the repeat block)

 

Select_ones:

 

I want to add a few notes on how select_ones behave.  In short, every time a field in your feature service uses a coded value domain, a new select_one type of question is created.  The coded value domains are matched by a corresponding XLSForm list in the choices worksheet.  You are free to reduce the number of choices in the list if  that makes sense for you. You can also change the labels in the list as you see fit. 

 

The values in the name column of the choices worksheet need to remain unchanged because otherwise, you will be adding data into your feature service outside the valid values set by your geodatabase domain.

 

Two more notes on geodabase domains: At the moment Survey123 Connect will ignore range value domains, although if you like you can model them as constraints in the XLSForm survey worksheet relatively easy.  Also, if your feature layer has subtypes and each subtype has been assigned a coded value domain, Connect will ignore the domains as well. If you really wanted to model that, you could setup cascading selects manually.

 

Other XLSForm columns and question types:  As far as the survey worksheet in your XLSForm, columns and question types other  than the ones described above will not be used. That is, the default, calculation, constraint, relevant columns etc will be empty.  Question types such as groups, notes or calculates will also never be present in the initial XLSForm design.  Now: you can, and should, use those extra columns and question types as appropriate to build a great data capture experience. For example:

 

  • You can use XLSForm groups to break down questions in your survey into logical sections. This simplifies and accelerates data capture without altering the original geodatabase schema in your feature layer.
  • You can leverage XLSForm notes and hints to provide field users with critical information to capture data more  efficiently. Similarly, notes and hints have  no effect on the geodatabase schema.
  • You can also use constraints, relevant statements as well as input masks, calculations and defaults to better control user input.
  • Appearances are also a great way to optimize the field data capture experience.

 

The initial XLSForm design is just that: a starting point. Take pride of your surveys and transform flat GIS data models into beautiful smart forms.

 

Once you are happy with your design, you will proceed to publish your survey. When publishing, no new feature service will be created for your form. Instead, the feature service you selected will be referenced by your survey.

 

Settings worksheet:

 

The settings worksheet of the initial XLSForm contains two key pieces of information:

 

  • submission_url: This column defines the url of the feature layer item in your ArcGIS Organization on which responses to your survey will be stored. This is the exact url of the feature layer you chose in the Create New Survey dialog as the seed of your survey.  Note that this is the url to the feature layer item, rather than the url of the feature service itslef.  In old versions of Survey123 we actually supported working directly with feature service urls, but when you let Survey123 Connect create your initial XLSForm design, the feature layer item url will always be used.  Right now, Survey123 supports feature service urls only for backwards compatibility. you should always use a url to the item!
  • form_id: This column defines the name of the layer in your feature service that your survey will hit.  By default, your initial  XLSForm design will always point to the first layer found in your feature service.

 

While you often may not want to change the submission_url, there is one use case where  changing the form_id is very handy.  Think for example, that you want to create a survey on top of a related standalone table.  For clarity, lets think of a feature service with parcels (polygons) and their related assessments.  You may not want to build a survey to edit the parcels, but you may want to create one to create the assessments. Typically, when you get the initial XLSForm design, you will have questions in your survey for all the fields in the parcels layer and then a repeat group, with all questions in the related assessment table. This is only because in a feature service, typically the layer with geometries (parcels in our case) goes first, and then the related tables.  To create your parcel assessment survey you will delete all questions  corresponding to the parcels layer from your survey and then take the name of your assessments repeat and put it into the form_id setting.  Finally you will delete the begin_repeat and end_repeat rows in your survey. 

 

To ensure that assessment data is properly attached to the corresponding parcel, it is important that in your survey you preserve a question to enter the unique identifier of the parcel. You will need to know what field in the child assessment table is used for the geodatabase relationship.

 

Limitations when working with Repeats in your XLSForm:

 

Mapping the ArcGIS geodatabase model into an XLSForm is possible to some extent, but there are some limitations to bear in mind.  In the case that you are working with existing feature services published from ArcGIS Desktop, there are chances that you could run into some situations where you simply cannot build a survey on top of a feature service, or at least not the exact survey that you had in mind.   The truth is that you can actually build surveys pretty much on top of any feature service, except when working with related tables.  Lets get into the details:

 

  • The XLSForm specification does not accept two questions in a single survey with the same name. This makes sense because otherwise you could not uniquely reference the question in your XLSForm expression.   When working with existing feature services this will be a problem when you have related tables AND fields in both tables with the same name.  In some cases this is not a real problem because you may not want to include the repeat on the survey. You may also be able to delete one of the questions referencing the duplicated field name (from either the main body of the survey, or from within the repeat). But this is not always the case...
  • The XLSForm specification does not accept spaces in the name of a repeat.  This is a problem when you have a related table in your feature layer that includes spaces in the name. If you encounter this you can do two things: One is obviously to delete the repeat group from your survey all together. If you cannot delete the repeat then you can ask the owner of the feature service to publish the feature service again, eliminating the space in the name of the related table. Of course this may not be possible in many cases, but I bring this up because changing the name of a layer in the feature layer does not imply changing the name of the table in the database.  All you need to do is to change the name of the layer in the table of contents in ArcGIS Desktop and publish again.
  • The XLSForm specification does not like spaces in the name column of the choices worksheet: This will be a problem when you have coded values in a domain that include spaces in them. Fortunately, you will only get a warning when validating your XLSForm in Connect. You will actually be able to publish your survey with no problem.  It is also technically possible to encounter coded values in domains that exceed the maximum length of 32 characters allowed by choice names.
  • Survey123 only lets you work with repeats when the underlying feature service has relationships that comply with the following:
    • Both the parent and child tables must have an indexed esriFieldTypeGlobalID field
    • The child table must use an indexed foreign key of type esriFieldTypeGUID against the GlobalID or GUID field in the parent table
    • If using attachments, the relationship of the attachments table with the parent table must be based on a GlobalID field.
    • The parent and child tables cannot be versioned.

 

If the conditions above are not met, you can still create a survey against your feature service, but you cannot include a repeat block in your survey. Technically, for Survey123 to be able to work with related records in a geodatabase, the feature service must have the supportsApplyEditsWithGlobalIds property set to true.  In our Survey123 help there are more details about this.

 

As you see, the limitations revolve around the use of repeats within your XLSForm.  If you do not have a repeat in your XLSForm, you can work against any feature layer.

 

 

Some tips and other things to be aware of:

 

  • In the old days of Survey123, people used to create the initial XLSForm designs by hand and from scratch. If you really understand the meaning of the submission_url and form_id XLSFom settings and you understand how to map fields in your feature service to XLSForm question types, you can do it.  Typically using the Create Survey from Feature service option as described above is the best way to go, but forcing yourself to build a survey from scratch on top of an existing feature service will be a good learning experience.
  • The dialog in Survey123 Connect listing all available feature services to be used as a seed for your survey will include line and polygon feature layers. Keep in mind that at the moment you cannot work against a layer with polygons and polylines, but you can work against any tables related to them.  Use the form_id setting to redirect your survey against the related table and eliminate questions outside the repeat.
  • Sharing can be problematic! Keep in mind that while you will always be the owner of the survey you are publishing, you may not be the owner of the feature service.  Unless you are the owner, you can't really control the sharing, so you will want to make sure that your feature service is made available to users that will access your survey.
  • Capabilities: Similarly, unless you are the owner of the feature service, you will have no control over the update, query, delete and add capabilities of the feature service. It is your responsibility to make sure your survey is consistent with the capabilities enabled in the service. For example, if the feature service does not support updates, then you should not enable the Inbox.
  • Hosted Feature Layer Views: You can create surveys on top of hosted feature layer views.  Views are a clever way to securely limit access to features and or columns in an existing feature service. Views also allow you to better control sharing of your feature service.  There are some scenarios where you may want to create surveys on top of views, rather than directly on top of the feature service.

Outcomes