Dates and Time in Survey123

[IsmaelChivite]

[Updated June 16, 2020]

[Updated September 18, 2023]

[Updated October 25, 2023]

Note: In October 2023, ArcGIS Online introduced new date and time field types. We will add support for these new fields in Survey123 in 2024.  This article will be updated accordingly once we have support for the new field types. Until then, everything below applies only to esriFieldTypeDate.

In this blog post I will describe how you can work with dates and time in Survey123 with XLSForm. I will also cover important concepts for you to understand how dates and time are modeled in ArcGIS.

Before we start

 

To best follow instructions and concepts in this blog, I suggest you open the Dates & Time survey sample in Survey123 Connect. If you have a first look at this sample survey, it will much easier for you to follow content in the blog.

1. Open Survey123 Connect
2. Tap on New Survey
3. Select the Samples category
4. Scroll until you see the Dates & Time survey sample or use the search function.
5. Create a new survey from this sample

Next, spend some time going over the survey. I suggest you explore it first in Connect and you also bring it into your phone or a web browser to see what it feels like. The structure of the blog pretty much follows the sample survey.

Basic Types and Appearances

The Basic Types and Appearances section of the survey sample explores the different ways in which a date or time question can be presented to the end user. Combinations of the question type and its corresponding appearances, will help you control the best user input style that fits the needs of your workflow.

The first question in this survey is of type date and it uses no appearance. A calendar control is shown for users to pick a date.  Note that the question shows the current date by default. This is accomplished by setting the default column in the XLSForm to today().

The following 3 questions illustrate the use of appearances: Year, Month-Year and Week-Number.

• Week-Number is the most exotic. Week numbers are often used in government and business for fiscal years and timekeeping. With this appearance, the calendar widgets will let the user pick a date and display the week number in the form.  You will also see the week numbers on the side as you select the date. The week numbers are calculated following the ISO-8601 standard.
• The Year and Year-Month appearances are handy when you simply want to present the user with a choice to pick a year, or a year-month.

As you exercise the calendar control, it may not be obvious, but you can actually tap and hold the arrows to quickly navigate back and forth in time.

• The DateTime and Time questions are shown at the end of this first section. Note that the questions automatically show the current date and time. This is accomplished by adding he now() function into the default column.

Understanding Dates and Time in ArcGIS

The Date, DateTime and Time types of questions as well as their corresponding XLSForm appearances, help you control the user experience that you want to present to the end-user capturing the data, but it is also important to understand how that date will be stored in ArcGIS.

User input to the Date and DateTime questions will be persisted in ArcGIS in a field of type Date (esriFieldTypeDate).  Despite the name of this field type, ArcGIS does not store just dates in Date fields: ArcGIS date values always include time as well. Hence, there is a perfect match between DateTime questions in Survey123 and ArcGIS date fields, but when it comes to date questions in Survey123, you need to understand that actually the time will also be stored.   But what time? Well, it depends:

• The time will be noon (12:00PM) if the user has manually selected a date, or if you set the date using the today() function.
• The time will vary if you set the date using the now() function. Now captures the current time, even if your date question does not show the time to the user in the form.

The screenshot below shows how you can configure a date field in your layer to display time. Note that in the Map Viewer, I had to open first the Fields dialog, then click on the date field and finally turn on the "Show time" option.   Once you do this, your popups will display time information!

Time values in Date and DateTime questions  are always persisted as Coordinated Universal Time (UTC). This is important when you have people working in different time zones. ArcGIS clients will translate the stored UTC values into your local time zone. The only exception is ArcGIS Pro. ArcGIS Pro will show the time in UTC (unless configured to do otherwise).

So far, I described how date and datetime questions behave. Lets talk next about Time questions, because they are special. For Time questions, Survey123 does not use the ArcGIS date type (which we know now that actually stores date and time). Survey123 time questions store values as a string. We did it this way, because otherwise you would get dates as well, and this could be confusing: Say you want to capture the time when a store opens.... you would not want the date in there...  Now, because time is stored as a string, there is no information regarding the time zone in it. As such, you need to keep in mind that there will be no time zone conversion.  Later in the survey, we will explore how to capture times differently.

Date and DateTime questions store values as date objects including time. Time questions store values as text.

Formatting Dates

All graphical controls in the Survey123 apps will honor your locale regional settings. For example, if your browser or smartphone's locale is set to Spanish, then dates will be formatted the Spanish way (which is the correct one, by the way... just kidding).  In the next screenshot I show what the calendar control will look like in a phone set with United States vs Spain regional settings. Note that when using the Spain regional settings, the day goes first, then the month and then the year, versus the US version where the month goes first then the day and then the year.

Additionally the format-date and format-date-time function allow you to convert date objects into strings.

In the following example, we  use the wsdatetime question to capture the date and time at which a water sample was taken. The wstime question calculate will be hidden from the end-user and is used to extract the time entered and persist it as a string in military format.  Note that the output of the wsdatetimequestion is stored in ArcGIS as a date object in UTC, versus the wstime which will be stored as a string using local time in your device.

type name label calculation

dateTime	wsdatetime	Sample Date/Time
calculate	wstime	Time	format-date(${wsdatetime},'%H:%M')

The qualifiers in the format-date function are as follows:

%Y	4 digit year
%y	2 digit year
%m	0-padded  month
%n	numeric month
%b	month name abbreviated
%d	0-padded day of month
%e	day of month
%H	0-padded hour (24-hr time)
%h	hour (24-hr time)
%M	0-padded minute
%S	0-padded second
%3	0-padded millisecond ticks (000-999)
%a	Three letter short text day

Date and Date-Time Defaults

Survey123 accepts three possible values in the XLSForm default column:

• today() This function returns the current date in your device. Typically you apply this function to questions of type date. The time part of this date will always by noon (12:00pm).
• now()  This function returns the current time in your device. Typically you apply this function to questions of type datetime.
• Finally, you can also apply a fixed date in ISO 8601 notation. For example: 2020-06-23T00:00:00-07:00.
  
  • 2020 for the year. Always 4 numbers.
  • 06 for the month. Always 2 numbers
  • 23 for the day. Always 2 numbers
  • T is simply a separator, for time. Always T.
  • 00 for the hour. Always 2 numbers
  • 00 for the minutes. Always 2 numbers
  • 00 for the seconds. Always 2 numbers
  • -07:00 is the UTC offset, which in this case is minus 7 hours.

Technically, you could also apply default value using Unix epoch notation (in milliseconds), but this would only work in the Survey123 field app. For consistency across the field and web apps, it is best to use ISO 8601 notation.

Constraints and Calculations

This section of the sample survey will be useful to understand how you use formulas with dates and date-times, so you can validate user input, as well as pre-compute date values. The questions in the survey are self-explanatory, The best is that you have a look at the XLSFile to better understand how things are done.

• Constraints: A common constraint you will want to enforce is one that ensures a date is in the past or the future. That is pretty straight-forward: .> now() will ensure the user input is in the future for example. If you want to reference another question in the survey, you can use something like this: ${birthdate}<today(). This will ensure that the birthday question has a value in the past.

In constraints you may be asked to ensure a value is always newer and older than a fixed date. The date function is very useful in these cases. For example, you can set a constraint such as . > date('2006-01-01') or ${manufacture_date} > date('2006-01-01').

If you need to work with time constraints, simply add the time component to the date() function using the ISO 8601 notation.

• Calculations: Calculations are easy too, as long as you understand that dates in the Survey123 app are encoded as numbers. When users pick a date or a date-time in the survey, all values will be presented in a human friendly way, but internally the application handles them as numbers that represent the total time elapsed since 1970. A positive number represents a date after 1970 and a negative number a date in the past.  As you can imagine those are pretty big numbers, but the math is simple once you understand how time is expressed. There are basically two formats for the elapsed time:

• Unix epoch is the elapsed time since 1970 expressed in milliseconds.

• Decimal date or French Revolutionary Time is the elapsed time since  1970 expressed in decimal days. That is, one day is 1.0, an hour is 1/24, a minute is 1/24/60, etc.   To learn more  about Decimal Time have a look at this 'How the French made a 10-hour day' article. You will love it!

	UNIX EPOCH	DECIMAL DATE
2 weeks into the future	now() + 14*24*60*60*1000	date(decimal-date-time(now()) +  14)
Calculate age from birthday	int((today() - ${birth_date}) div (1000*24*60*60* 365.25))	int((decimal-date-time(today()) - decimal-date-time(${birth_date})) div 365.25)
Minutes spent for lunch	int(${LunchEnd} - ${LunchStart}) div (1000*60)	int((decimal-date-time(${lunchends}) - decimal-date-time(${lunchstarts})) * 24*60)
15 minutes from now	now() + 15*60*1000	date-time(decimal-date-time(now()) +  0.0104166675)

0.0104166675 is equal to  (1 (day) / 24 (hours) / 60 (minutes) ) times 15 (minutes).

I recommend you work with decimal date format, because then your XLSForm expressions will work both in the field and web apps. The Survey123 web app ONLY supports decimal time.

If you plan to use date expressions in the web app, use the decimal-date-time function.

If you ever need to populate a Survey123 date or dateTime question using a UNIX epoch number, you can use an expression like this:

type	name	label	calculation
text	epoch	Epoch (milliseconds)
dateTime	mydate	The Date	date-time(int(${epoch}) div 86400000)

 

Calculates and Relevants

This section in the sample survey is very similar to the previous one, but shows some other techniques that may be handy. For example, note how a calculate question is used to get the Lunch time computed, and then that value is used in other expressions throughout the survey.   A calculate is a type of question that will let you keep the result of a calculation without having to show it to the user in the survey. 

You will remember that questions of type time are ultimately stored as strings in ArcGIS, however, as far as calculations go, you will treat them as numbers. That is actually pretty handy because otherwise, you would not be able to do much with them!

Other ways to capture Time

The last section in the sample survey shows other techniques to capture time.

• The 'Time from a list' example should not come as a surprise. A select_one type of question with minimal appearance is a nice way to present a list of hours to choose from.
• Then there are some other examples illustrating how to use regular expressions in constraints as well as input masks.  I will go into the details here, so nothing is missed.
  • First, an example to capture military type with a period. For example: 11:35.  Note the use of an input mask here. The input mask automatically adds the period (:), so the user does not have to worry about that. The value of the input mask is 99:99 meaning that we expect to digits followed by a period and two more digits. You will see that the constraint column checks to make sure that the hours do not go beyond 23 and the minutes do not go beyond 59.  Note: input masks are NOT supported in the Survey123 web app.
  • In the second example, we want to capture military time, but without the period. For example: 1135. In this case, I want to highlight a couple of things. First, note that I am not using an input mask like before. It is not needed, because we can set the type of question as integer (to ensure that only numbers can be typed) and also set the the maximum length of the field to 4. This is done through the bind::esri:fieldLength column. This will do two things: will ensure that only integers with 4 digits can be entered into the Feature Service, and most importantly, will limit the user input in the Survey, to 4 numbers.  I still added a regular expression in the constraint to ensure that proper values within the 24 hour day are entered.
  • In the third example, I want to capture military time including the time zone. For example 1445Z. Z is the time zone (ZULU, which represents UTC). In 1445G, G is for Golf and represents UTC + 7.  That would be 1445 Golf Time. How awesome is that! More details on this crazy business with Time Zone Abbreviations here. Again in this case I added a mask and complemented it with a regular expression in the Constraint.

If you are not familiar with input masks in Survey123 I recommend you have a look at the Esri custom columns help topic. Regular expressions are tricky to put together, but you can find loads of samples in the web.

Start and End

So far, we have been looking at date and time questions that were visible in the survey. It is for this reason that we cared about the user experience (types of questions, defaults and appearances) as well as the validation of inputs (input masks, constraints).  Now, in Survey123 there are a couple of handy question types that will capture time without the end user knowing. These questions are start and end.  When added to your survey, like in the Sample you are looking at, they will not show in the survey to the end-user, but they will populate data in the Feature Service, that you can see from the survey123 web site or a web map.

Start and end will capture the exact date and time at which the survey was initially opened by the end user, and flagged as done. For you to see how this works, the best is that you load the survey in your phone, complete a couple of surveys and submit the data.

The start and end values are useful if you want to understand how much time people in the field take to complete surveys. To calculate the duration of the survey, you will need to download the data in FileGeodatabase or Shapefile format and calculate a new field with ArcGIS for Desktop. This ArcWatch article describes how you can calculate the duration between two date fields.

On top of start and end, which are optional, you will always get the Creation and Last Edit date and time for every row you add from Survey123 for ArcGIS. It is important to highlight here that the Creation and Last Edit columns are handled through ArcGIS Editor Tracking. The help has a good topic on Editor Tracking in case you are not familiar with it. Do not worry about  enabling Editor Tracking in Survey123, because it is always enabled by default. This feature looks at the time when edits are made in the Feature Service not when the data is necessarily being captured or changed in the field. For example, if someone is working in the field while disconnected from the network, the start and end field will truly reflect the time when the user is working on the survey. The Creation and Edit Times will reflect the time when the data was actually submitted: May be immediately after data capture, or may be a couple days later while back in the office.

This was a long post, but I hope you find it useful.