Find new entries when streaming real-time data

Question

Hi everyone,

I am streaming real-time data using GeoEvent Server and then visualize them on the map using stream service. The response format is JSON and I am using "Poll an External Website for JSON" input connector. I have a GeoFence in the area and I want to send a notification when a new point falls within the GeoFence. Each record has a unique ID. Is there any way that I can bring over only new records (For example by comparing IDs with previous records)?

Thank you very much for your help.

Anonymous User · Answer

Hi RJ,

Thank you very much for your complete explanation RJ Sunderman.

RJSunderman · Answer

Hello Hossein –

Every event record received by an inbound connector is generally atomic, by which I mean the adapter/transport used to implement the connector has no knowledge of event records previously received or knowledge of event records about to be received. The same goes for the nodes (e.g. the processors and filters) in a GeoEvent Service.

There are some exceptions, with their own limitations. A filter configured with a spatial operation ENTER, for example, needs to know if the geometry of an event with a TRACK_ID previously observed was outside or disjoint so that it knows the current event record's geometry, which is now inside, should evaluate as having entered the polygon used to model the geofence.

Likewise, a TrackGap Detector processor needs to maintain a cache of TRACK_ID values it has previously observed so that it knows when a report from a given asset is expected but has not been received. There are other examples, but I won't go into those here. What you need to take away from this that GeoEvent Server - fundamentally - does not hold onto or cache data it is processing. Doing so runs counter to the objective of ingesting, processing, and disseminating data as quickly as possible at the highest possible volume.

Since you stipulated you only want to notify when the geometry from a new tracked asset, as identified by a unique TRACK_ID, has entered an area ... I'm assuming that if a tracked asset leaves and later re-enters the area you do not particularly care. The best way best I can think of, then, to notify

(a) that an event record's geometry intersects or is "inside" a geofence and

(b) not re-notify that an event record is still inside the area, or later re-enters the area

would be to save any event record you've determined intersects your geofence as a feature record. The feature record's schema will need to have an additional field hasBeenNotified whose value is initially set to, say, zero.

Part of the trick to making this work is https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2018/10/03/using-a-partial-structure-or-schema-to-update-feature-records?sr=search&searchId=52457f48-aa43-4b7c-b83f-c070908bf928&searchIndex=6‌. If you deliberately omit the hasBeenNotified attribute from the GeoEvent Definition used to add/update feature records as real-time data is received from a sensor, and configure the feature service to apply a default value for hasBeenNotified when no value is specified, then only new feature records will have the hasBeenNotified attribute assigned to zero. It's important to recognize that this zero value is set as a default by the ArcGIS Server Feature Service, not by GeoEvent Server. Event records you receive from the real-time sensor are primarily driving updates into existing feature records, using the unique TRACK_ID for values such as date/time reported and location, and not touching the hasBeenNotified attribute value. You are relying on a partial GeoEvent Definition to leave the hasBeenNofified attribute value of existing feature records unmodified.

Then, when your other input polls the feature service for feature records whose hasBeenNotified attribute value is zero, it uses a different GeoEvent Definition – one that does include the hasBeenNotified attribute – along with a Field Calculator to specifically update the hasBeenNotified attribute of the feature record it just polled with the value '1'. The GeoEvent Service you configure for notification, then, is not only pushing out notifications ... it's recording the fact that a notification has been made into the feature record set from which it collects feature records which require notification.

Hope this information helps –
RJ