Skip navigation
All Places > Survey123 for ArcGIS > Blog > 2017 > December
2017

Survey123 is an incredible app from Esri that makes it really easy to configure data capture surveys. I've previously looked at how to implement a custom Other field and  how to make survey results easier to analyse, or symbolise, when a survey includes the Other functionality.  These methods use configuration options in the Survey123 Connect for ArcGIS desktop application. Supposing you’ve created your survey using the web designer, or perhaps you are happy with your existing surveys and don’t want to change them? In this blog I explore how to work with results that include the Other functionality, after the survey has been published.

If you're not aware, the Other functionality allows users to enter an Other value as an answer to a multiple choice question if the answer they need is not in the list. This has been implemented for the question What is the property type?:

This functionality is awesome, but the Other value that a user inputs is assigned to a new field in the attribute table. This means the answers for one question are spread over two fields:

Arcade can help us with this! Arcade is a cross-platform expression language for ArcGIS and, since  June 2017, we can use Arcade in web map pop ups. So we can manipulate the contents of pop ups and, in this instance, display the correct entry for the question What is the property type?

This blog assumes you have published a survey using the Survey123  Web Designer or  Connect, with at least one Single Choice (web designer) or select_one (Connect) question with Other enabled. You can do this using the  standard method or the alternative method I shared before.

Using Arcade with Other

1)  Add the Survey123 data to a web map

2)  Navigate to the Configure Pop Up pane > under Attribute Expressions click Add

3)  Specify a name for the Attribute Expression

4)  Now we need to form the expression. In simple terms, if the answer to a question is ‘other’, then ignore it and give us the alternate answer supplied instead. In Arcade, this translates into:

IIf($feature.fieldname == "other", $feature.fieldname_other, DomainName($feature,"fieldname"))

For the question in this example; ‘What is the property type’ is the field alias and ‘PropertyType’ is the field name so the expression looks like:

IIf($feature.PropertyType == "other", $feature.PropertyType_other, DomainName($feature,"PropertyType"))
Attribute Expression.PNG

Click Test to ensure the expression works correctly. When you’re happy, click OK to close the window.

Note: Arcade expressions are case sensitive, 'other' might be 'Other' or another phrase depending on your survey question.

5)  In the Configure Pop Up pane, choose A list of field attributes for the Displayoption > click Configure Attributes. The Attribute

Expression created in the previous step will appear at the bottom of the list. Enable this attribute, reorder and turn off the two

fields it is replacing:

CC_Nov17 Configure Pop Up.png

 

Click OK to close the Configure Pop Up pane

6. Click on a feature to see the results!

Arcade Pop Up.PNG

Tips

  • You can find the field names from the Globals section in the Attribute Expression window
  • If you are referencing a field’s domain in the expression, you need to use the domain code and not the name. You can find this in the Attribute Expression window by clicking on the little pencil beside the field name under Globals.
  • If you created your survey using the Survey123 Web Designer, please note that the field names usually are field_0, field_0_other, field_1 etc.
  • Under Globals, fields that have domains include extra information:

 

Field Code or Name.png

If you use the Code then domain codes will be returned in the pop up, whereas if you use DomainName($feature,”PropertyType”) the domain names will be returned instead and will look better.

  • Attribute Expressions can also be used in Custom Attribute Displays too, have a go!

Using Arcade in this way means no changes are made to the information at a database level but how the information is relayed to end users is improved!

Note, if you need some help setting up a Survey123 survey, check out the help  videos and  pages.

This post was originally shared on the Esri UK blog: GeoXchange — Esri UK. It's also a valuable source for posts on other ArcGIS topics by other Esri employees, check it out!

Digital cameras typically embed metadata at the point of capture into image files. EXIF (Exchangeable Image File Format) is the most widely adopted standard for encoding this metadata within the photos you take.  Some common data fields include the camera make and model, its serial number, the date and time of image capture, the shutter speed, the aperture, the ISO speed setting...  If a GPS is available to the camera, EXIF will also include the location from which the photo was taken and also the direction.

 

All the above information can be critical so we added quite extensive support for EXIF data within Survey123 for ArcGIS.

 

Preserving and capturing EXIF data with your photos.

 

The Survey123 field app is designed to preserve and capture EXIF data whenever possible.  When a new image question is added to your survey, you can use the Survey123 app to either browse for an existing photo, or use your device's camera to capture a new one.

 

If you browse for an existing photo, the Survey123 app will preserve any existing EXIF data in case at some point you want to download the photo from ArcGIS and extract the EXIF metadata.  This is important particularly when you use high-end cameras that write EXIF tags of your interest, or when you simply are handed a photo from someone else.

 

If you choose to capture a new photo with the Survey123 field app, not only EXIF metadata will be populated according to the  capabilities of your device, but also a handy GPS button will be available so you can refresh the GPS coordinates from your device right when you take the photo. The reason why this is important is because the GPS in your device may not have fixed the best possible location when you shoot the photo. By using the GPS button in the Survey123's app camera widget you can help getting the best possible location into your EXIF's metadata.

 

Looking into basic EXIF metadata tags

 

The Survey123 website has built-in reporting capabilities that let you look at basic EXIF metadata in photos submitted. Simply go into the data tab, and expand the Individual Survey Response report for a particular record. Simply click on photos within the report to access basic EXIF metadata such as the coordinates and direction where the photo was taken, the device used to capture the photo and the date and time.

 

 

Extracting EXIF metadata using the pulldata() XLSForm function

 

My favorite feature in Survey123 in regards with EXIF is the ability to write expressions in your XLSForm that will automatically extract EXIF tags from your photos so you can build data validation rules in your smart form and even store this EXIF tags as attributes in your features.  This means that later on you will be able to easily query your EXIF metadata within ArcGIS without any post-processing.  A classic example is extracting the GPSImageDirection tag into an attribute, so you can later use it to rotate your symbols on a map, or the GpsLongitude and GpsLatitude tags to later map the location of the photos. Other handy tags include ImageWidth, ImageLength and ExtendedDigitalZoomRatio which give you information about the size of the image as well as the optical zoom level used (if any) when the photo was captured, and of course the ImageDateTime tag as well

 

You can extract the EXIF tags using the pulldata() function, which we extended specifically for this purpose. The syntax is as follows:


pulldata("@exif", ${image}, "GpsImageDirection")

 

The first parameter indicates that the pulldata() function will be used to extract EXIF metadata. The second parameter indicates the question in your XLSForm with the image. The third parameter defines the name of the EXIF tag to be extracted.

 

There are over 300 tags in the EXIF specification, although there is no guarantee that all of them will be populated. It varies a lot depending on the device.  The table below shows the most common ones.

 

EXIF GeotagsEXIF Basic Camera  and Image TagsExtended Tags
GpsVersionId
GpsLatitudeRef
GpsLatitude
GpsLongitudeRef
GpsLongitude
GpsAltitudeRef
GpsAltitude
GpsTimeStamp
GpsSatellites
GpsStatus
GpsMeasureMode
GpsDop
GpsSpeedRef
GpsSpeed
GpsTrackRef
GpsTrack
GpsImageDirectionRef
GpsImageDirection
GpsMapDatum
GpsDestLatitudeRef
GpsDestLatitude
GpsDestLongitudeRef
GpsDestLongitude
GpsDestBearingRef
GpsDestBearing
GpsDestDistanceRef
GpsDestDistance
GpsProcessingMethod
GpsAreaInformation
GpsDateStamp
GpsDifferential
ImageWidth
ImageLength
ImageBitsPerSample
ImageCompression
ImagePhotometricInterpretation
ImageOrientation
ImageSamplesPerPixel
ImagePlanarConfiguration
ImageYCbCrSubSampling
ImageXResolution
ImageYResolution
ImageResolutionUnit
ImageStripOffsets
ImageRowsPerStrip
ImageStripByteCounts
ImageTransferFunction
ImageWhitePoint
ImagePrimaryChromaciticies
ImageYCbCrCoefficients
ImageYCbCrPosition
ImageReferenceBlackWhite
ImageDateTime
ImageDescription
ImageMake
ImageModel
ImageSoftware
ImageArtist
ImageCopyright
ImageXPTitle
ImageXPComment
ImageXPAuthor
ImageXPKeywords
ImageXPSubject
ImageRating
ImageRatingPercent
ImageLensSpecification
ImageLensMake
ImageLensModel
ImageLensSerialNumber
ExtendedExifVersion
ExtendedFlashPixVersion
ExtendedColorSpace
ExtendedComponentsConfiguration
ExtendedCompressedBitsPerPixel
ExtendedPixelXDimension
ExtendedPixelYDimension
ExtendedMakerNote
ExtendedUserComment
ExtendedRelatedSoundFile
ExtendedDateTimeOriginal
ExtendedDateTimeDigitized
ExtendedSubSecTime
ExtendedSubSecTimeOriginal
ExtendedSubSecTimeDigitized
ExtendedImageUniqueId
ExtendedExposureTime
ExtendedFNumber
ExtendedExposureProgram
ExtendedSpectralSensitivity
ExtendedISOSpeedRatings
ExtendedOecf
ExtendedShutterSpeedValue
ExtendedApertureValue
ExtendedBrightnessValue
ExtendedExposureBiasValue
ExtendedMaxApertureValue
ExtendedSubjectDistance
ExtendedMeteringMode
ExtendedLightSource
ExtendedFlash
ExtendedFocalLength
ExtendedSubjectArea
ExtendedFlashEnergy
ExtendedSpatialFrequencyResponse
ExtendedFocalPlaneXResolution
ExtendedFocalPlaneYResolution
ExtendedFocalPlaneResolutionUnit
ExtendedSubjectLocation
ExtendedExposureIndex
ExtendedSensingMethod
ExtendedFileSource
ExtendedSceneType
ExtendedCfaPattern
ExtendedCustomRendered
ExtendedExposureMode
ExtendedWhiteBalance
ExtendedDigitalZoomRatio
ExtendedFocalLengthIn35mmFilm
ExtendedSceneCaptureType
ExtendedGainControl
ExtendedContrast
ExtendedSaturation
ExtendedSharpness
ExtendedDeviceSettingDescription
ExtendedSubjectDistanceRange

 

To get you started quickly with the use of the pulldata() function, I recommend you have a look at the EXIF XLSForm sample included with Survey123 Connect for ArcGIS.

 

  1. Open Survey123 Connect for ArcGIS
  2. Click on New Survey
  3. Find the EXIF survey in the Samples category
  4. Set a title to your new survey and create it
  5. The list of tags is located in the choices worksheet

 

In most cases I see the pulldata() function being used to simply persist EXIF tags from photos as attributes in your feature service, but you can also use them to build input data validation logic. For example, you can compare the location of a geopoint question in your survey with the location of a photo and prevent the user from submitting if the distance is bigger than a particular threshold. You could also display a warning if the user attempts to use a photo that was taken a long time ago etc.  This is were you get creative to get the best out of Survey123...

Survey123 is a very popular ArcGIS App for the Field and I'm always looking for new ways to work with it - to make sure it's delivering value for my users. Previously I've looked at applying a custom Other field and in this blog I investigate making survey results easier to analyse, or symbolise, when a survey includes the Other functionality.

If you're not aware, the Other functionality allows users to enter an Other value as an answer to a multiple choice question, if the answer they need is not in the list. This has been implemented for the question What is the property type?:


This functionality is awesome, but the Other value that a user inputs is assigned to a new field in the attribute table. This means it can be hard to analyse and query this question later on because the answers for one question are spread over two fields:

Blog 1 v2.png

Here's a method to collate these answers to one field, making it easier for end users to view and query the data. So instead of the answers looking like the above example, they will look like this:

Blog 3 v2.PNG

If you need some help setting up a Survey123 survey, check out the help  videos and  pages. This blog assumes you have a survey set up using the Survey123 Connect for ArcGIS desktop application with at least one  select_one type question.

The blog is split into 2 sections to cover different methods used to implement Other

  • If you are using the or_other method, look at Section 1.
  • If you are manually adding Other as a choice option, look at Section 2 
  • If you are working with choice names that include spaces, step 5 in Section 2 covers this.

Please note: the methods described below will only work in the Survey123 survey. If you edit the data in the ArcGIS Online map viewer or in Collector for ArcGIS, the changes won't automatically be reflected. 

Section 1

1)  Browse to the select_one question you have set up with or_other:

In the background, Survey123 creates a new field to hold the Other value entered by the end user. The name of this field is usually the name of the select_one field followed by ‘_other’. In the example above, the new field created for the is called PropertyType_other (you can also find this field name by looking in the Schema Preview in the Survey123 Connect for ArcGIS desktop application).

2)  Create a new text question under the select_one question (this is for testing, later this field will be of type hidden). Name and label it something like this:

3)  Now you need to create an if statement to determine what value is written to this new field. This if statement will be entered into the calculation field. The if statement for this survey will be:

if(${PropertyType}='other', ${PropertyType_other}, ${PropertyType})

It’s important that 'other' is all lowercase as this is what Survey123 will recognise in this case.

In simple terms: if the PropertyType question answer is ‘other’, then assign the PropertyType_other value to this field. Otherwise assign the PropertyType value to this field.

The survey will look like this:

4)  Save the Excel spreadsheet and preview the functionality in Survey123 Connect:

Or_other collate.gif

5)  When you’re happy with the survey, head back to Excel and change the type of question to hidden for PropertyTypeCollated:

This means that this question will be hidden from end users in the Survey123 app, but will be available for you later on. So you can display it in a pop up or use it as a field for analysis/querying.

**Hidden fields are not currently supported in the Survey123 website (http://survey123.arcgis.com) but has been added as an enhancement request. 

6)  Your survey is now ready to be published and used on a mobile device.

Section 2

1)  Browse to the select_one question you have set up with Other as a choice option:

Blog 19.png

2)  Create a new text question under the select_one question. This will be the text field to hold the Other - SurveyType answer. Name and label it something like this:

3)  Now you need to create an if statement to determine what value is written to this new field. This if statement will be entered into the calculation field. The if statement for this survey will be:

if(${SurveyType}='Other', ${SurveyTypeOther}, ${SurveyType})

This Other is looking at the name column in the choice list.
In simple terms: if the SurveyType question answer is ‘Other’, then assign the SurveyTypeOther value to this field. Otherwise assign the SurveyType value to this field.
The survey will look like this:
4)  Save the Excel spreadsheet and preview the functionality in Survey123 Connect:
Other collate.gif
5)  You may notice above that if you choose Health and Safety as the answer, the SurveyTypeCollated field has the text HealthandSafety. This is because Survey123 pulls the text from the name column and not the label column:
Blog 12.png

We can expand the current if statement to compensate for choices that include spaces/characters by nesting the if statements:

if(${SurveyType}='HealthandSafety', 'Health and Safety', if(${SurveyType}='Other', ${SurveyTypeOther}, ${SurveyType}))

In simple terms: if the SurveyType question answer is ‘HealthandSafety’, then assign ‘Health and Safety’ to this field, if the SurveyType question answer is ‘Other’, then assign the SurveyTypeOther value to this field. Otherwise assign the SurveyType value to this field.

More information about nested if statements can be found here but please note that Survey123 does not need the statements to begin with '='.

6)  Save the Excel spreadsheet and preview the functionality in Survey123 Connect.

7)  When you’re happy with the survey, head back to Excel and change the type of question to hidden for SurveyTypeCollated:

This means that this question will be hidden from end users in the Survey123 app but will be available for you later on. so you can display it in a pop up or use it as a field for analysis/querying.

**Hidden fields are not currently supported in the Survey123 website (http://survey123.arcgis.com) but has been added as an enhancement request. 

8)  Your survey is now ready to be published and used on a mobile device.

Collating answers to one field will make creating queries, filters, applying symbology and running analysis easier!

 

This post was originally shared on the Esri UK blog: GeoXchange — Esri UK. It's also a valuable source for posts on other ArcGIS topics by other Esri employees, check it out!

We often get requests from people who want to update a file in the media folder of a Survey123 Form item but do not want to have to re-publish their survey every time they make a change to the file. For example, the file could be a large CSV containing address information that is constantly updated from another source, and is part of an external choice list in your survey. Or you may have several image thumbnails or audio files used in questions in your survey that need to be updated regularly. Currently the only way to update these files in your survey is to update (copy and paste) the file into the media folder for that survey on your computer, and re-publish the survey via Connect.

 

 

In this blog, I will describe a technique for updating the media folder using a Python script and how to use Windows Task Scheduler to run the Python script at a regular interval. The script will connect to ArcGIS Online or Portal, download the survey Form item, extract the zip file to a folder, copy the updated media folder files from a known location into the extracted folders, zip the Form item back up again, upload it to ArcGIS Online or Portal, resulting in the Form item being updated. Users of the field app will need to delete the survey from their gallery in the field app on the device, re-download the survey from the Download Surveys menu, open the survey and they will have access to the updated survey and associated media files.

 

Below are instructions so you can configure the Python script with your own survey and media files.

 

Configuring the Python script

 

The script requires Python 2.7 or above to run and the 'ArcREST' Python module to be installed. If you have ArcGIS Pro or ArcGIS Desktop installed, then Python 2.7 or above should be installed, however you will likely not have the ‘ArcREST’ module. Either way, you can simply install Python and then the 'ArcREST' module as follows:

 

 

Once you have Python and 'ArcREST' installed, you need to configure the script as follows:

 

  1. Download the updateMediaFolder.py script from ArcGIS Online, extract from the zip file and save to your computer. Copy the updateMediaFolder.py script to your working directory (download_folder) where you will execute the Python script from; this folder will be where the temporary folders and files will be created to update the media files in the Form item each time the script is run.
  2. Edit and save the “User Parameters” section in the updateMediaFolder.py script as follows:

itemID = ItemID of the Form item you want to update

download_folder = Local folder where the Form data will be downloaded (include a / at end of path)

updated_file = Updated file name (ie MyList.csv)

source_loc = Folder path of the updated file (include a / at end of path)

portal_url = URL address of your ArcGIS portal

credentials = ArcGIS user credentials to authenticate against the portal (NB: case sensitive)

security_type = ArcGIS portal security type (LDAP, NTLM, OAuth, Portal, PKI)

 

Once the above user parameters have been updated in the script, save your changes and you are ready to go. Simply execute the script and the Form item of your survey will be updated.

 

Please note that updating the itemsets.csv file directly in the Form item via this script does not update the values in the external_choices sheet of the associated xls workbook. It only updates the generated CSV file in the media folder. If you open the survey in Connect again, or download the survey to Connect from AGOL after you have manually updated the itemsets.csv with this script, there will be a difference between the external_choices sheet in xls workbook (original values) and the itemsets.csv (new values). If you re-publish the survey via Connect again without making any changes to external_choices sheet, the itemsets.csv will be replaced with the original values that are in the external_choices sheet.

 

If you want to schedule this script to run regularly to update your Form item, the following may assist in getting that set up.

 

Scheduling the script

 

Windows operating systems include a simple utility called Task Scheduler. Once you have determined how often your media files change and at what interval you need to update the media folder, you can repeat its execution using Task Scheduler. For example, every 60 minutes. Obviously, the computer where you have setup the task will need to be running all the time and set to run regardless of who is logged in.

 

The configuration of tasks in the scheduler are pretty much self-explanatory, but here are some specific instructions that can save you some time:

 

  1. General: Check the option to run with highest privileges and set the task to run even if you are not logged-in.
  2. Triggers: If you want to quickly test your task, you can simply select your task in the gallery and then hit Run in the Selected Item panel on the right. When configuring the task for real, I suggest you select the startup trigger and that you also configure the task to run indefinitely.
  3. Actions: You will need to be particularly careful with this one. The Program/Script setting needs to point to the Python executable.
  • If using Python from ArcGIS Pro, it will be under the Pro installation directory. For example: C: \Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe"
  • If using your own copy of Python, refer to the installation directory of Python.

 

You also need to indicate and pass the location of your updateMediaFolder.py script as an argument. Do not forget to include the .py extension in the path. Lastly, set the 'Start in' property so it points to the directory in which you are storing the updateMediaFolder.py file.

 

 

Beyond the basics

 

The technique described in this blog post is simple but effective and will enable you to update files in the media folder of your survey without having to re-publish via Connect. However, please note that all care should be taken to ensure the files you are updating with this script have the exact name as the files in the originally published media folder. Also, the files should have the same format, with the same field names as the original files (in the case of CSV files). Only rows of data should be updated or additional rows added. If you want to change the format of the files (rename or add columns) you should update the files via Connect in the media folder, update the survey xls form, and then re-publish the survey to ensure the changes do not break anything.

 

It is recommended to test the script on a backup copy of your survey and ensure the survey can be downloaded and updated in the field app, checking that the external choice lists and/or media items are working as expected. This should be confirmed before running the script and updating the media folder and files on your real survey that is currently in use by other Survey123 field app users.

Hi all,

 

I wanted to let everyone know that we've updated the Survey123 Beta Program with a new version of Survey123.  This release includes several fixes as well as initial testing support for a new functionality - webhooks. Webhooks are a configuration option that will allow Survey123 to accomplish a second action when submitting a survey; for example, sending an e-mail with some of the submitted information.  If you're interested in trying this out, please enroll in the Early Adopter Community!

 

The Esri Early Adopter Community enables our users to to test early versions of upcoming releases of Esri applications.  Survey123 publishes beta versions of the software during every development cycle to test and try out new features.  If you're interested in trying out new functionality and fixes, please enroll using the link above!

Survey123 is an easy way to create new surveys or replace existing paper-based surveys because of the flexibility of its functionality.

One example of this flexibility is the ability to allow users to enter an  Other value as an answer to a multiple choice question if the answer they need is not in the list. This has been implemented for the question What is the property type?:

In the background, Survey123 creates a new field to hold this Other value entered by the end user. The default name of this field is the name of the select_one field followed by ‘_other’, and the alias for this field is ‘Specify other.’.

This means you aren’t able to define your own alias to the question in the survey and, if this is repeated functionality, you can end up with multiple fields with the alias ‘Specify other.’.

I'm going to show you an alternative method of setting up an Other option using the question What type of survey is this?. This post assumes you have a survey set up using the Survey123 Connect for ArcGIS desktop application, with at least one  select_one type question. If you need some help setting up a Survey123 survey, check out the help  videos and  pages.

1.      Find the select_one question you want to implement Other for:

2.      Navigate to the corresponding list for this question in the Choices sheet:
3.      Add Other to the SurveyType list:
Blog 12.png
You have the flexibility of giving a more descriptive option for users here like:
4.      Create a new text question under the select_one question. This will be the text field to hold the Other answer for SurveyType. Name and label it something like this:
5.      Next we need to make this question optional, so it only appears if the end user selects Other for the SurveyType question. For this example, enter ${SurveyType}='Other' in the relevant column:
6.      Save the Excel spreadsheet and try out the functionality in Survey123 Connect for ArcGIS.
7.      When you’re happy with your survey, publish it and see it in action!
This post was originally shared on the Esri UK blog: GeoXchange — Esri UK. It's also a valuable source for posts on other ArcGIS topics by other Esri employees, check it out!