ichivite-esristaff

Dates and Time in Survey123 for ArcGIS

Blog Post created by ichivite-esristaff Employee on Apr 17, 2016

[Updated September 9, 2018]

 

In this blog post I will describe how you can work with dates and time in Survey123 for ArcGIS. I will also cover important concepts for you to understand how dates and time are modeled in ArcGIS and some tips.

 

Before we start

 

In this occasion, simply reading a blog post is not going to cut it. It is critical that you start Survey123 Connect, so you can use the Dates & Time survey sample as we proceed through this post.

 

  1. Open Survey123 Connect for ArcGIS (Latest versions for Windows or Mac)
  2. Tap on New Survey
  3. Select the Samples category
  4. Scroll until you see the Dates & Time survey sample
  5. Create a new survey from this sample

 

 

Next, spend some time going over the survey. I suggest you explore it first in Connect and you also bring it into your phone or tablet so you can see what it feels like in a mobile device. Next, lets go together through the different sections in this survey.

 

Basic Types and Appearances

 

The Basic Types and Appearances section explores the most essential concepts:

 

  • There are three types of questions that matter: Date for capturing dates, DateTime for capturing a date and a time, and Time, for just capturing time.Dates04.png
  • The first question in this survey is of type date.  Note that by default is showing todays' date. This is accomplished by using today() in the default column of your spreadsheet.
  • The following questions illustrate the use of appearances: Year, Month-Year and Week-Number.
    • Week-Number is the most exotic. Week numbers are often used in government and business for fiscal years and timekeeping. With this appearance, the calendar widgets will let you pick a date and display the week number in the form.  You will also see the week numbers on the side as you select the date. The week numbers are calculated following the ISO-8601 standard.
    • The Year and Year-Month appearances are handy when you simply want to present the user with a choice to pick a year, or a year-month.
  • As you exercise the calendar control, it may not be obvious, but you can actually tap and hold on the arrows to quickly navigate back and forth in time.

 

Dates02.png

  • I left the DateTime and Time questions at the end of this first section. Note that in this case I used the now() expression for the default column. Understanding the difference between now() and today() is important. But to make sense of the difference you need to understand first how your Date, DateTime and Time questions are going to be stored in the ArcGIS Feature Service.

 

Understanding Dates and Time in ArcGIS

 

The Date, DateTime and Time types of questions in Survey123 help you control the user experience that you want to present to the end-user capturing the data.  

 

Date and DateTime questions: User input to the Date and DateTime questions will be persisted in ArcGIS in a field of type Date.  Despite the name of this field type, ArcGIS does not store just dates in Date fields. In fact, ArcGIS dates always include time as well. Hence, there is a perfect match between DateTime questions in Survey123 for ArcGIS and ArcGIS date fields, but when it comes to date questions in Survey123, you need to understand that actually the time will also be stored.   But what time? If using the today() default, the time will be set to the beginning of the day (12:00AM). The beginning of the day will also be used if the user picks a date in the calendar widget.  When using a DateTime type of question, the time stored will be that chosen by the user. You can also preset the time by using the now() expression in the default column of a time question.

 

It is a bit tricky to get your time to show in ArcGIS, so that deserves an explanation too. The thing is that by default, your web maps in ArcGIS will hide the time... even though it is actually always stored. The screenshot below shows how you can configure your popup to display time. If the show time check box is disabled, do not panic: Change the format to a numeric date format, and it will switch on.

 

Dates03.png

 

Time values in Date and DateTime questions  are always persisted as Coordinated Universal Time (UTC). This is important when you have people working in different time zones. ArcGIS clients will translate the stored UTC values into your local time zone, so they make sense to you.

 

Time questions: So far, I described how date and datetime questions behave. Lets talk next about Time questions, because they are special. For Time questions, Survey123 does not use the ArcGIS date type (which we know now that actually stores date and time). Time questions are actually going to store values as a string. We did it this way, because otherwise you would get dates as well, and this could be confusing: Say you want to capture the time when a store opens.... you would not want the date in there.  Now, because time is stored as a string, there is no information regarding the time zone in it. As such, you need to keep in mind that there will be no time zone translation.  Later in the survey, we will explore how to capture times differently.

 

Date and DateTime questions are by default mapped to fields of type Date. Time questions, however, are mapped to fields of type string.

 

Formatting Dates

 

All graphical controls in the Survey123 app will honor your locale regional settings. For example, if your smartphone's locale is set to Spanish, then dates will be formatted the Spanish way (which is the correct one, by the way... just kidding).  In the next screenshot I show what the calendar control will look like in a phone set with United States vs Spain regional settings. Note that when using the Spain regional settings, the day goes first, then the month and then the year, versus the US version where the month goes first then the day and then the year.

 

 

Additionally the format-date function allows you to convert date objects into strings.

 

In the following example, we  use the WSDateTime question to capture the Date & Time at which a water sample was taken. The WSTime question is hidden from the end-user and is used to extract the Time entered and persist it as a string in military format.  Note that the output of the WSDateTime question is stored in ArcGIS as a date object in UTC, versus the WSTime which will be stored as a string using local time in your device.

 

typenamelabelcalculation
dateTimeWSDateTimeSample Date/Time
hiddenWSTimeTimeformat-date(${WSDateTime},'%H:%M')

 

The qualifiers in the format-date function are as follows:

 

Header 1Header 2
%Y4 digit year
%y2 digit year
%m0-padded  month
%nnumeric month
%bmonth name abbreviated
%d0-padded day of month
%eday of month
%H0-padded hour (24-hr time)
%hhour (24-hr time)
%M0-padded minute
%S0-padded second
$30-padded millisecond ticks (000-999)
%aThree letter short text day

 

Defaults

 

Survey123 accepts three possible values in the XLSForm default column:

  • today() This function returns the current date in your device. Typically you apply this function to questions of type date.
  • now()  This function returns the current time in your device. Typically you apply this function to questions of type datetime.
  • Finally, you can also apply a fixed date or time, using UNIX time notation.That is, you cannot just type December 14, 1974 and hope that Survey123 will parse it. You actually need to type the equivalent UNIX time value.  I like using https://www.epochconverter.com/ to get the UNIX time values. So for example, for December 13, 1974 the UNIX time (GMT) is 156124800.

 

Constraints and Calculations

 

This section of the survey will be useful to understand how you use formulas with dates and date-times, so you can validate user input, as well as pre-compute date values. The questions in the survey are self-explanatory, The best is that you have a look at the XLSFile to better understand how things are done.

 

Dates05.png

  • Constraints: A common constraint you will want to enforce is one that ensures a date is in the past or the future, or the past. That is pretty straight-forward: .> now() will ensure the user input is in the future for example. If you want to reference another question in the survey, you can use something like this: ${birthdate}<today(). This will ensure that the birthday question has a value in the past.

 

In constraints you may be asked to ensure a value is always newer and older than a fixed date. The date function is very useful in these cases. For example, you can set a constraint such as . > date('2006-01-01') or ${manufacture_date} > date('2006-01-01').

 

  • Calculations: Calculations are easy too, as long as you understand that dates in the Survey123 app are encoded as numbers. When users pick a date or a date-time in the survey, all values will be presented in a human friendly way, but internally the application is actually handling them as numbers that represent the total time elapsed since 1970. A positive number represents a date after 1970 and a negative number a date in the past.  As you can imagine those are pretty big numbers, but the math is simple once you understand how time is expressed. There are basically two formats for the elapsed time:

 

  • EPOCH or  Unix Time is the elapsed time since 1970 expressed in milliseconds.
  • Decimal Date or French Revolutionary Time is the elapsed time since  1970 expressed in Decimal days. Now, you may be wondering what Decimal Time is. Well, this is a French Invention. To learn more  about Decimal Time have a look at this 'How the French made a 10-hour day' article. You will love it!

 

EPOCH FormatDecimal Date Format
2 weeks into the futurenow() + 14*24*60*60*1000date(decimal-date-time(now()) +  14)
Calculate age from birthdayint((today() - ${birth_date}) div (1000*24*60*60* 365.25))

int((decimal-date-time(today()) - decimal-date-time(${birth_date})) div 365.25)

Minutes spent for lunchint(${LunchEnd} - ${LunchStart}) div (1000*60)

int((decimal-date-time(${lunchends}) - decimal-date-time(${lunchstarts})) * 24*60)

15 minutes from nownow() + 15*60*1000

date-time(decimal-date-time(now()) +  0.0104166675)

 

If you choose EPOCH or Decimal time for your calculations is a bit what makes  you feel more comfortable, but you should also know that if you go the French way, your surveys will work well in web browsers as well as in the native Survey123 app. If you go with the EPOCH approach, your calculations will only work in the Survey123 native app.

 

 

Calculates and Relevants

 

This section in the sample survey is very similar to the previous one, but shows some other techniques that may be handy. For example, note how a Calculate question is used to get the Lunch time computed, and then that value is used in other expressions throughout the survey.   A calculate is a type of question that will let you keep the result of a calculation without having to show it to the user in the survey.  Once kept, you can use the value in the calculate question as many times as you like, without having to repeat the computation.

 

You will remember that the questions of type Time are ultimately stored as strings in ArcGIS, however, as far as calculations is concerned, you will operate with them as numbers. That is actually pretty handy because otherwise, you would not be able to do much with them.

 

Other ways to capture Time

 

The last section in the Date & Time sample survey shows other techniques to capture time.

 

  • The list example should not come as a surprise. A select_one type of question with minimal appearance is a nice way to present a list of hours to choose from.
  • Then there are some other examples illustrating how to use Regular Expressions in constraints as well as input masks.  I will go into the details here, so nothing is missed.
    • First, an example to capture military type with a period. For example: 11:35.  Note the use of an input mask here. The input mask automatically adds the period (:), so the user does not have to worry about that. The value of the input mask is 99:99 meaning that we expect to digits followed by a period and two more digits. You will see that the constraint column checks to make sure that the hours do not go beyond 23 and the minutes do not go beyond 59.
    • In the second example, we want to capture military time, but without he period. For example: 1135. In this case, I want to highlight a couple of things. First, note that I am not using an input mask like before. It is not needed, because we can set the type of question as integer (to ensure that only numbers can be typed) and also set the the maximum length of the field to 4. This is done through the bind::esri:fieldLength column. This will do two things: will ensure that only integers with 4 digits can be entered into the Feature Service, and most importantly, will limit the user input in the Survey, to 4 numbers.  I still added a regular expression in the constraint to ensure that proper values within the 24 hour day are entered.
    • In the third example, I want to capture military time including the time zone. For example 1445Z. Z is the time zone (ZULU, which represents UTC). In 1445G, G is for Golf and represents UTC + 7.  That would be 1445 Golf Time. How awesome is that! More details on this crazy business with Time Zone Abbreviations here. Again in this case I added a mask and complemented it with a regular expression in the Constraint.

 

If you are not familiar with input masks in Survey123 I recommend you have a look at the Esri custom columns help topic. Regular expressions are tricky to put together, but you can find loads of samples .

 

Start and End

 

So far, we have been looking at date and time questions that were visible in the survey. It is for this reason that we cared about the user experience (types of questions, defaults and appearances) as well as the validation of inputs (input masks, constraints).  Now, in Survey123 for ArcGIS there are a couple of handy types of questions that will capture time without the end user knowing. These questions are Start and End.  When added to your survey, like in the Sample you are looking at, they will not show in the survey to the end-user, but they will populate data in the Feature Service, that you can see from the survey123 web site or a web map.

 

Start and End will capture the exact date and time at which the survey was initially opened by the end user, and flagged as done. For you to see how this works, the best is that you load the survey in your phone, complete a couple of surveys and submit the data.

 

The Start and End values are extremely useful if you want to understand how much time people in the field take to complete surveys. To calculate the duration of the survey, you will need to download the data in FileGeodatabase or Shapefile format and calculate a new field with ArcGIS for Desktop. This ArcWatch article describes how you can calculate the duration between the Start and End fields.

 

On top of Start and End, which are optional, you will always get the Creation and Last Edit date and time for every row you add from Survey123 for ArcGIS. It is important to highlight here that the Creation and Last Edit columns are handled through ArcGIS Editor Tracking. The help has a good topic on Editor Tracking in case you are not familiar with it. Do not worry about  enabling Editor Tracking in Survey123, because it is always enabled by default. This feature looks at the time when edits are made in the Feature Service not when the data is necessarily being captured or changed in the field. For example, if someone is working in the field while disconnected from the network, the Start and End field will truly reflect the time when the user is working on the survey. The Creation and Edit Times will reflect the time when the data was actually submitted: May be immediately after data capture, or may be a couple days later while back in the office.

 

This was a long post, but I hope you find it useful.

Outcomes