nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Timestamps received from a sensor feed display differently in GeoEvent Sampler, ArcGIS REST Services queries, and ArcGIS Pro

RJSunderman

This post was developed by Esri staff with the purpose of modeling real customer questions and positioning them with answers in a way that complements our users’ search processes.

We use Verizon Connect to track vehicles in our van pool in Cheyenne Wyoming. The clocks on our vehicles are set to Mountain Standard Time (MST) which is -7 hours relative to UTC.

When I use the GeoEvent Sampler to look at data from my input I see the date and time in MST, which is what I want. The data also appears to be in MST when I look at output I’ve written to a CSV file. Here are a couple of screenshots of what I’m seeing:

When I look at feature records in the database the date and time look like they’ve been switched to an integer. When I open the feature record’s attribute table in ArcGIS Pro all of the values are offset 7 hours.

So, data coming into GeoEvent Server is MST but is somehow being changed when written to the database to switch it to UTC? Why is this happening and how do I fix it?

Encontre mais publicações marcadas com

Accepted answers

RJSunderman

To answer your question we should first take a look at the raw data GeoEvent Server is receiving. The date/time values associated with sensor data are highly adaptable and can be represented many different ways. Your question mentions receiving data from Verizon Connect, so let’s look at some sample data from that sort of data feed.

The illustration above left shows a sample of raw JSON data typical of what Verizon Connect might send. Note the UpdateUTC data value is sent as a 13-digit epoch value in milliseconds. When a GeoEvent Server input receives data in this format and adapts it as a Date the epoch value is assumed to represent a Coordinated Universal Time (UTC) value, not a local time value such as Mountain Standard Time (MST). So to clarify a point in your question, data is not received as MST values. It is received as an epoch long integer value and adapted, first for processing as a Date data type, and later for display as a string.

The second illustration, above right, shows an online date/time utility I often use to convert epoch long integer values to human-readable string representations of a given date and time. Note that the strings constructed by the utility specify an offset from Greenwich Mean Time (GMT). GMT and UTC never change for Daylight Saving Time (DST) and are sometimes used interchangeably even though GMT is technically a time zone and UTC is a time standard.

A display string constructed from the epoch 1676597835000 can specify a time zone. For example, an application might construct a string to display a date and time as February 17th 1:37:15 AM (GMT) or display a local time February 16th 6:37:15 PM (MST). Note that when the epoch converter displays a local time value a time zone offset is included in the string.

Client applications generally determine how they want to construct and display string values representing a date and time. GeoEvent Server uses epoch long integer values for its Date values when processing data, and at times converts the Date into a human-readable string for display in the server’s local time zone.

To your original question, the UpdateUTC data value is not changed when written to the database. The Date value in a geodatabase feature record is always going to be an epoch long integer value consistent with the ArcGIS REST API. Verizon Connect sends the date/time of a vehicle position report as a UTC value, and that is how the value is being stored in the database.

Why a date/time value displayed by the GeoEvent Sampler appears different, for example, than the same value written out to a CSV file or a JSON file is a question of data adaptation and string construction. Rather than displaying the actual epoch long integer value of a Date (such as 1676597835000) GeoEvent Sampler constructs and displays a string representation of the Date using your server’s locale to determine an appropriate time zone. That is why you see the time zone MST included in the GeoEvent Sampler’s displayed string "Thu Feb 16 18:37:15 MST 2023".

When you configure a Write to a CSV File output you choose whether date/time values are written out as ISO 8601 formatted values or as strings in a custom format. The Write to a JSON File output, on the other hand, cannot be configured to write out a human-readable string. When reviewing the JSON output file in a text editor you will see the actual epoch long integer value of a Date.

I’ll include some examples in the comments below to illustrate how you might expect different client applications to construct display strings from epoch long integer values stored in a geodatabase feature record. But to answer the last part of your question, how do you fix what you are seeing, that depends entirely on what you are trying to do.

The epoch long integer underlying a Date cannot express a time zone or an offset from UTC the way a String value can. You could configure a Field Calculator or Field Mapper with an expression that either adds or subtracts a number of hours from a Date value, but I really do not recommend this. If you fudge a Date value to shift it from UTC into a local time zone you risk a client application downstream, possibly one outside the ArcGIS Enterprise, constructing a string from an epoch value it assumes is a UTC value and displaying a string which appears incorrect.

Suppose, for example, you are using an application like SQL Server Management Studio (SSMS). When the application retrieves a Date whose value you have explicitly offset, it will likely assume the value it retrieved is a UTC value, use your server’s locale to determine an appropriate time zone, and apply the temporal offset a second time. The Date value you explicitly offset by a number of hours as part of your event record processing in GeoEvent Server now appears incorrect when viewed using SSMS.

For the simple reason that a database server can be in any time zone, and web client applications that access the data may not be in the same time zone as the server, the recommended best practice is to keep the ArcGIS REST API default and allow feature services to maintain Date values as epoch long integer values in the assumed UTC standard. If you want to calculate and store a local representation of a Date value, calculate the value as a String. You can use the toString( ) function in a Field Mapper, for example, to do this.

A String value is a literal string and won’t ever be manipulated to change its value. This might be ideal for displaying attribute values in a web map pop-up, but you cannot use attributes of type String to configure something like the time slider in ArcGIS Pro. Instead of computing a String value, or adding/subtracting a number of milliseconds from a Date value, the best approach working with ArcGIS Pro would be to configure its feature layer properties to apply a time zone offset to Date values it retrieves from a geodatabase. As you zoom, pan, and potentially change a map’s temporal extent to see more or fewer features, the date and time strings displayed by ArcGIS Pro should reflect local time rather than a UTC time.

All comments

RJSunderman

Data from my sensor feed reports timestamp values in seconds rather than milliseconds. When these timestamps are converted to string values they display as dates in 1970. How do I fix this?

Some data feeds report date/time values as 10-digit epoch values measured in seconds. GeoEvent Server uses 13-digit millisecond values consistent with the ArcGIS REST API, so you have to multiply by 1000 to scale the value from seconds to milliseconds before writing it out to a geodatabase

To prevent an epoch such as 1676597835 from displaying as "Tue Jan 20 02:43:17 MST 1970":

Allow your input to adapt the 10-digit epoch value as either a Date or a Long integer value
Use a Field Calculator or Field Mapper to scale the ReportedDT attribute value from seconds to milliseconds
Use a toDate( ) function to explicitly cast the arithmetic result from an implicit Long to a Date

Example: toDate(ReportedDT * 1000)

RJSunderman

Data from my sensor feed reports the date and time of each data record in separate fields. How do I combine these into a single Date attribute?

Use a Field Calculator or Field Mapper to combine the date and time values into a string which GeoEvent Server can interpret as a Date. Your goal is to construct a single ISO 8601 formatted string that contains both the date and time values which you can then cast from String to Date.

For this example assume that ReportedDate and ReportedTime hold the values "02/16/2023" and "18:37:15" respectively. Assume that the date and time are reported as local values, so a dynamic UTC offset should be applied to avoid errors when clocks are adjusted for Daylight Savings twice each year.

The string you are calculating for this example should be: 2023-02-16T18:37:15-00:00

Allow your input to adapt the ReportedDate and ReportedTime as separate String value
Use substring( ) functions to slice values from these strings and append them to a new String
Use a toDate( ) function to cast the constructed String value to a Date
Use the currentOffsetUTC( ) function to offset the constructed Date from a local to a UTC standard value
Write the result into a new attribute field whose data type is Date

Example:

The expression in the example presented above has been formatted within a code block for readability.
The expression should be entered as a single line of text into a Field Calculator or Field Mapper processor.

You can copy/paste the text shown below:

toDate( substring( ReportedDate, 6, 10 ) + '-' + substring( ReportedDate, 0, 2 ) + '-' + substring( ReportedDate, 3, 5 ) + 'T' + substring( ReportedTime, 0, 2 ) + ':' + substring( ReportedTime, 3, 5 ) + ':' + substring( ReportedTime, 6, 8 ) + '-00:00' ) - currentOffsetUTC()

Each substring( ) function in the example slices a few characters out of the expression’s target field. For example, substring(ReportedDate, 6, 10) slices the four characters '2023' from the reported date '02/16/2023'.

The several + in the expression append substrings to one another. The '-' in the expression are literal dash characters. Your goal is to construct the string: 2023-02-16T18:37:15-00:00

The currentOffsetUTC( ) function uses your machine’s locale to determine a millisecond offset from local time to UTC taking current Daylight Savings adjustments into account. The example said to assume the date and time are reported as local values so we subtract the current UTC offset to shift the constructed Date to a UTC standard consistent with the string’s time zone designation (the '-00:00' we appended to the constructed ISO 8601 String). The UTC offset for regions in North America is negative as these time zones are several hours behind Greenwich Mean Time (GMT). Subtracting a negative value from the date/time effectively adds a number of hours to the value pushing it to a UTC standard time.

Appending a time zone designation '-00:00' is a better approach than hard-coding an offset such as '-07:00' into the constructed string. Whatever constant you choose as the offset for a local time zone is likely only accurate during certain months of the year. Adjusting the date/time by adding the dynamic UTC offset takes your server's current Daylight Savings observation into account for you.

Reference: https://www.w3.org/TR/NOTE-datetime

RJSunderman

Data from my sensor feed reports timestamp values which look like the ISO 8601 format, but Date values converted to a string display with an offset hours ahead of the local time reported in the feed. Why is this happening?

Sometimes the date/time strings sent from a data provider appear to follow ISO 8601 formatting rules, but they are missing a time zone designator or the literal ‘T’ between the date and time values. To answer your question we should first confirm that the raw data being received actually follows the ISO 8601 standard.

For this example assume:
a) The sensor feed's specification indicates that ReportedDT is sent as a local time value
b) The ReportedDT string value "2023-02-10T11:45:00" is being received

The date/time value being sent by the data feed does not include a time zone designation. You can apply a dynamic UTC offset based on your server’s observed time zone as part of a Field Calculator or Field Mapper expression to fix this.

Allow your input to adapt the ReportedDT string value as a Date
Use currentOffsetUTC( ) to offset the adapted Date from its local value to a UTC standard value
Cast the result of the calculation to a Date (the arithmetic calculation returns a Long, not a Date)

Example: toDate(ReportedDT - currentOffsetUTC())

The feed's value really ought to include a time zone designator to indicate in which time zone the date/time is being reported. It would be the feed’s responsibility to automatically adjust the time zone designation in data it sends when clocks change for Daylight Savings twice each year. If no offset were being applied the time zone designation should be '-00:00' to indicate the date/time is a UTC value.

The data provider’s specification, in this example, supposedly says that the date and time values are consistent with a local time zone. Applications which adapt an ISO 8601 formatted string which is missing a time zone designator are free to assume a time zone. GeoEvent Server assumes the date/time is a UTC value when adapting an ISO 8601 formatted string. The current UTC offset is therefore subtracted from the reported date/time to shift the value from what the feed specification says is a local time to the UTC standard assumed by GeoEvent Server. The value returned by the currentOffsetUTC( ) function is negative for areas in North America. Subtracting a negative value effectively adds a number of hours to the Date shifting it forward to a UTC standard time.

You might think to fix the ReportedDT value by adapting it as a String and then appending a time zone designator yourself to the string. If you do, twice each year when clocks are adjusted for Daylight Savings, the time zone designator you append to the received string would require update. So it is better to use the dynamic UTC offset provided by the currentOffsetUTC( ) function in this case.

Remember, Date values in a geodatabase are maintained as epoch long integer values which by convention are assumed to be UTC values. As a best practice you want to write UTC values to your geodatabase and allow client applications to determine how they want to construct a string representation of an assumed UTC value they retrieve as part of a feature record

Reference: https://www.w3.org/TR/NOTE-datetime

Participe da Conversa

Membros conectados podem postar, seguir atualizações e mais. Novo aqui? Registre uma conta gratuita.

Links Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.