Filter out Time Anomalies

StevenBirney1 — Fri, 20 Nov 2015 15:51:34 GMT

I'm using the Sierra Wireless GX440 and the RAP adaptor to track vehicles. Sometimes I'm receiving events with an odd date/time stamp of 11/30/1999 with varying times. I also receive date/time stamps of 11/30/2099 with varying times. I've tried filtering out these anomalies a couple different ways with no success. For example using Epoch milliseconds "TimeStamp >= 1448034146471" (11/20/2015 15:42:26) and "TimeStamp <= 4099698000000" (11/30/2099). Using that method no events come through. I've also tried "TimeStamp >= 2015y 11M 19d" with no success.

Is there any other way to filter these odd dates out?

Re: Filter out Time Anomalies

RJSunderman — Fri, 22 Jan 2016 23:41:35 GMT

Hello Steven -

My apologies that your question went so long without an answer or reply.

I don't have a Sierra Wireless setup I can use to truly replicate your workflow, but I can assume a data provider is sending invalid date/time values and choose another way of ingesting the data.

I configured a 'Receive JSON on a REST Endpoint' and sent the following three records into a GeoEvent Service

[

{"TrackID":"AA-1234", "DT":"11/30/1999 12:34:56"},

{"TrackID":"BB-3456", "DT":"01/22/2016 12:34:56"},

{"TrackID":"CC-5678", "DT":"11/30/2099 12:34:56"}

]

The date/time values are received as string values, but I made sure my GeoEvent Definition specifies that the attribute values be handled as Date. I used a stream stream service to broadcast the unfiltered output so that I could see the epoch milliseconds value GeoEvent was handling:

{"attributes":{"TrackID":"CC-5678","DT":4099754096000}}
{"attributes":{"TrackID":"AA-1234","DT":943994096000}}
{"attributes":{"TrackID":"BB-3456","DT":1453494896000}}

I've been told that attempting to configure a Filter with any of the following expressions will not work:

DT >= ${EffectiveFrom}	Comparing an event attribute against another field of type Date
DT >= 2/15/2015 13:26:45	Comparing an event attribute against a String literal
TIME_START <= ${MyEventTime}	Using a GeoTag to identify a field to compare against another field

But using the date/time algebra you tried (e.g. TimeStamp >= 2015y 11M 19d) should have worked.

I tested the following expression in a simple Filter element: DT >= 2010y 01M 01d

That expression successfully filtered/removed the invalid '1999' value.

When I changed my filter to have two expressions (with a logical AND): DT >= 2010y 01M 01d AND DT <= 2030y 01M 01d

both invalid dates were filter/removed. Only the event with the TrackID "BB-3456" passed through to my stream service output.

You may want to check your GeoEvent Definition to make sure that your TimeStamp field is being handled as a Date. You could also use a File/JSON output or a stream service output to confirm the date/time value associated with the events you are receiving.

I confirmed that a Filter element's expression will not accept a literal integer value representing epoch milliseconds. A Field Calculator processor performs implicit data type conversions as part of its expression evaluation (which is why we can add/subtract some number of milliseconds from a Date value) ... but a Filter is limited to the date/time algebra I used above.

The product team has an issue open in the product backlog ("The date format does not work on filter in GeoEvent Services - Issue #581"). If applying filters to date/time values continues to be an issue for you and the date/time algebra is not a workable solution, please submit an incident with Esri Technical Support and request the incident be escalated and linked to the existing issue in the product team's backlog.

Hope this information was helpful -

topic Filter out Time Anomalies in ArcGIS GeoEvent Server Questions

Filter out Time Anomalies

Re: Filter out Time Anomalies