Date Filter not working

Question

Hi All,

I am using the following input connector : Watch a Folder for New CSV files and output connector of Add Feature in GeoEvent Server.

I am now trying to configure a filter based on date field however, it's not working and the in and out count matches. Ideally, I am looking to get records which have been acquired during the day time like records between 5 AM to 5 PM.

Following is the screenshot of expression I have used :

I am very new to GeoEvent so not sure what exactly am I missing here.

RJSunderman · Answer

@wizgis -- It's not a problem that the date and time are part of the same data value being ingested. We actually prefer that. The GeoEvent Definition specifies that the input should be adapting data values as a Date. The fact that acquisitionTime is expressed as an ISO 8601 standard value helps guarantee GeoEvent Server's input will be able to adapt the value it receives and create a Date.

I'm not sure that you can configure a Filter literally with $RECEIVED_TIME, but you should be able to configure it with the name of the attribute field (e.g. acquisitionTime) as you show in  your illustrations. That is how I tested the configuration I included in my previous response as a screenshot.

I did perceive what looks like a typo in the one screenshot you shared:

I'm sure you took care of that extra bit of punctuation, but I saw it in your screenshot, so I thought I'd mention it.

One thing to check is the actual value of the Date your GeoEvent Server adapts from the ISO 8601 value received by the input. Adaption of an ISO 8601 value assumes UTC. Given the input and output below:

You'll note that the value 1669849214000 can be represented as either of the following strings:
  o  Wed Nov 30 15:00:14 PST 2022
  o  Wed Nov 30 23:00:14 GMT 2022

A string representation of the epoch value 1669849214000 must be constructed in order to extract values for hh, mm, and ss to plug into the filter's algebra. The filter, I think, is choosing to construct the string in the server's local time zone -- which for me is the first string above, showing PST.

Obviously if you are expecting the filter to only keep values between 05:00 and 17:00 and the filter is using strings constructed in local time, then 1669849214000 and 1669852814000 both satisfy the filter. These two epoch values are 15:00:14 and 16:00:14 (local, PST). The values 1669856414000 and 1669860014000 are both greater than 17:00 once the epochs are converted to strings in the local time zone. The on-line utility https://www.epochconverter.com can help you convert epoch values to string values in both GMT and your local time zone.

At this point if you are still having trouble configuring the filter and working with the date/time value conversions, I think it would be helpful for you to open an incident with Esri Technical Support so that someone can work with you, your data, and correlate behavior for your time zone.

Hope the above helps --
RJ

wizgis · Answer

@RJSunderman Yes, I came across the blog of how to create temporal filter and hence, I was using $ReceivedTime expression though am not sure if this is the right way to go.

I did try using the AND operator however, still running into the same issue.

I also tried using a Date field with AND operator however, in this case none of the records came through.

I am wondering could this happen because of the fact that in the date field there is date as well as time and do I need a separate field just for time. ]Here is snippet of data

I am also attaching GeoEvent definition being used.

RJSunderman · Answer

It looks to me like you're using what I refer to as the "algebraic expressions" Morakot describes in his blog How to Create Temporal Filter in GeoEvent

The way I read the filter you screenshot above, any event record whose received time is greater-than 1500 hours will be allowed to pass through as well as any event record whose received time is less than 1600 hours. All event records will satisfy that conditional.

I think this is more like what you want:

When you double-click to edit the filter you need to make sure you apply the logical condition AND to the two expressions you have entered as the filter's criteria: