Understanding GeoEvent Definitions

[RJSunderman]

Understanding what a GeoEvent Definition is and how they are created is important for understanding the real-time analytics, filtering, and processing being performed by a GeoEvent Service.

In the Making Features Come Alive exercise in Module 2 of the Introduction to GeoEvent tutorial, you are introduced to the Field Mapper Processor. One of your first experiences with GeoEvent Definitions will probably be when using a GeoEvent Service to add or update features in a feature service’s feature layer. Mapping a GeoEvent to a schema consistent with the feature layer before sending an event to an output to add/update features is a recommended best practice.

You should think of every GeoEvent as having an associated GeoEvent Definition. An input connector does not receive GeoEvents. The connector has a transport which receives and hands-off a byte stream to the connector’s adapter. The adapter needs a GeoEvent Definition in order to instantiate different data values and place the data into different fields within a data structure. It’s like the adapter is sorting mail -- this field is a String, that field is a Date, that other field is a Long integer,… etc. The structure of a GeoEvent is specified by a GeoEvent Definition.

Several input connectors available out-of-the-box with the GeoEvent Extension use adapters which allow you to determine whether the adapter should create a GeoEvent Definition. The adapter does this by taking the first event received, examining the data values, and then making a best-first-guess at what the GeoEvent Definition should look like. If a GeoEvent Definition already exists with the name you specified, the adapter will use (not update) this existing GeoEvent Definition.

GeoEvent Definitions have owners. When an adapter creates a GeoEvent Definition for you the owner will look something like: auto-generated/com.esri.ges.adapter.inbound.JSON

If you copy an existing GeoEvent Definition, import a GeoEvent Definition from a feature service, or click 'New GeoEvent Definition' to create one from scratch, the GeoEvent Definition owner will typically be displayed as either arcgis (10.2.x) or admin (10.3.x) depending on how you logged in to ArcGIS GeoEvent Manager.

Only two GeoEvent Definitions come defined out-of-the-box. The incident GeoEvent Definition owner is identified as com.esri.ges.processor/IncidentDetector/xxxx. You can see this if you click to examine the incident GeoEvent Definition. The TrackGap GeoEvent Definition is the other GeoEvent Definition that is defined out-of-the-box. If you click to examine this GeoEvent Definition you will see that its owner is com.esri.ges.processor/TrackGapDetector/xxxx. These owner strings tell you which out-of-the-box processor created the GeoEvent Definitions; the xxxx identifies the GeoEvent Extension release.

Sometimes a processor will modify an event’s structure or schema by either adding or removing a field. When a processor does this, it will create a new GeoEvent Definition. The Field Calculator Processor is a good example; you can configure this processor to place its calculated value into a new field. The Field Enricher Processor is another example; it is taking data from a feature service's table and adding new fields to the event it is processing.

When a processor creates a GeoEvent Definition for you, you are typically required to enter the name the processor should use for the GeoEvent Definition it will create.

It can be important to understand when a processor creates what I have referred to as a managed GeoEvent Definition. A processor will not create a managed GeoEvent Definition until it actually receives a GeoEvent. You have probably discovered that you cannot, for example, configure a Field Mapper Processor which follows a Field Enricher until the Field Enricher has received and processed an event. Once the Field Enricher receives a GeoEvent, it will process the event and create the necessary GeoEvent Definition. Only then can you edit your GeoEvent Service to configure your Field Mapper.

Since processors own the GeoEvent Definitions they create, any change you make to the GeoEvent Service which incorporates the processor will trigger the processor to delete its managed GeoEvent Definition when you publish the GeoEvent Service to save your changes. For this reason, you should be aware of when GeoEvent Definitions are created, when they are deleted, and when they are available for reference by another processor or output connector.

A few recommendations to take away from this quick discussion:

1. Every GeoEvent has an associated GeoEvent Definition. You should become familiar with when GeoEvent Definitions are created, which components create them, and why.
   
2. Don’t leave an input connector configured to create GeoEvent Definitions for you. When configuring an input connector to create a GeoEvent Definition, it is a recommended best practice to copy the GeoEvent Definition generated by the input connector’s adapter and then reconfigure your input to use the copy of the GeoEvent Definition.
   
3. Processors can create GeoEvent Definitions. The GeoEvent Definition associated with an event received by a processor is not necessarily going to be the GeoEvent Definition associated with the GeoEvent which comes out of a processor.
   
4. Always use a Field Mapper Processor to prepare a GeoEvent’s schema to match a feature service’s feature layer when adding/updating features. You might think you understand the structure of an event you are sending to an output – but it is a best practice to use a Field Mapper and make the schema mapping explicit.
   
5. Processors may delete GeoEvent Definitions they created. If you configured a Field Mapper “downstream” from a Field Calculator or Field Enricher, for example, the “upstream” processor may delete a GeoEvent Definition referenced by the Field Mapper. As long as you do not double-click to edit the Field Mapper’s configuration you can trust the processor “upstream” will re-create the GeoEvent Definition referenced by the Field Mapper as event data is received by the processor, before GeoEvents are received by the Field Mapper.
   
6. When publishing a stream service, never refer to a GeoEvent Definition which has been generated for you by an input connector’s adapter or a GeoEvent Definition created for you by a processor. Make a copy of any such GeoEvent Definition(s) and publish the stream service to reference a GeoEvent Definition which you own.

[ThibautDelaere1]

Very useful post! Thanks!

So, a GeoEvent processor can only delete GeoEvent Definitions they own? Because I see definitions disappear that I made myself. See my question:Updating GeoEvent Services causes GeoEvent Definitions to disappear​

[RJSunderman]

Thanks Thibaut ... yes, I found a bug logged against the 10.3 product release and replied to the thread: Updating GeoEvent Services causes GeoEvent Definitions to disappear

- RJ

[JayFlorey]

Thank you for this post. It answered a question I had. Now I just have to be patient and wait for a new event to update my Field Mapper.

Jay

[RJSunderman]

That can be a challenge, when data received from a real-time feed comes with significant delays.

There should be nothing to prevent you from including a second input into a GeoEvent Service which you can use to drive your own events into your event processing flow -- either to test the analytics you are developing do what you expect, or in this case, force a processor to re-create a GeoEvent Definition 'just-in-time' for you to get on with your work.

I usually use the 'Receive JSON on a REST Endpoint' input to do this.

I'll use my Chrome Poster plug-in to HTTP/POST a block of JSON which matches what my actual real-time data feed would normally provide. That way I can "simulate" a single event being sent. Sometimes when my actual real-time input is noisy I'll stop it and use my 'Receive JSON on a REST Endpoint' input until I've completed the changes I want to make to the GeoEvent Service, then restart the actual real-time data feed input.

If you are unfamiliar with using a browser plug-in to POST content to a URL, take a look at Module 6 in the GeoEvent product introduction tutorial. By the time I got to spatial processors I had finally grown weary of using the TCP/Text input and the GeoEvent Simulator and began using a the  'Receive JSON on a REST Endpoint' input to POST data to GeoEvent.

One trick when using this input: Create the input and save it, then edit it to have GeoEvent Manager reveal to you the URL created and hosted by GeoEvent. This is the URL to which you will want to POST your JSON data.

Hope this information helps -

RJ

[JayFlorey]

Hi RJ,

Thanks for the tip. It will be useful moving forward on implementation. I have also found the output to a JSON file very useful in creating samples to use for testing both GEP and ArcGIS Server feature services.

Regards,

Jay