ArcGIS GeoEvent Server Blog - Page 11

Understanding what a GeoEvent Definition is and how they are created is important for understanding the real-time analytics, filtering, and processing being performed by a GeoEvent Service.

In the Making Features Come Alive exercise in Module 2 of the Introduction to GeoEvent tutorial, you are introduced to the Field Mapper Processor. One of your first experiences with GeoEvent Definitions will probably be when using a GeoEvent Service to add or update features in a feature service’s feature layer. Mapping a GeoEvent to a schema consistent with the feature layer before sending an event to an output to add/update features is a recommended best practice.

You should think of every GeoEvent as having an associated GeoEvent Definition. An input connector does not receive GeoEvents. The connector has a transport which receives and hands-off a byte stream to the connector’s adapter. The adapter needs a GeoEvent Definition in order to instantiate different data values and place the data into different fields within a data structure. It’s like the adapter is sorting mail -- this field is a String, that field is a Date, that other field is a Long integer,… etc. The structure of a GeoEvent is specified by a GeoEvent Definition.

Several input connectors available out-of-the-box with the GeoEvent Extension use adapters which allow you to determine whether the adapter should create a GeoEvent Definition. The adapter does this by taking the first event received, examining the data values, and then making a best-first-guess at what the GeoEvent Definition should look like. If a GeoEvent Definition already exists with the name you specified, the adapter will use (not update) this existing GeoEvent Definition.

GeoEvent Definitions have owners. When an adapter creates a GeoEvent Definition for you the owner will look something like: auto-generated/com.esri.ges.adapter.inbound.JSON

If you copy an existing GeoEvent Definition, import a GeoEvent Definition from a feature service, or click 'New GeoEvent Definition' to create one from scratch, the GeoEvent Definition owner will typically be displayed as either arcgis (10.2.x) or admin (10.3.x) depending on how you logged in to ArcGIS GeoEvent Manager.

Only two GeoEvent Definitions come defined out-of-the-box. The incident GeoEvent Definition owner is identified as com.esri.ges.processor/IncidentDetector/xxxx. You can see this if you click to examine the incident GeoEvent Definition. The TrackGap GeoEvent Definition is the other GeoEvent Definition that is defined out-of-the-box. If you click to examine this GeoEvent Definition you will see that its owner is com.esri.ges.processor/TrackGapDetector/xxxx. These owner strings tell you which out-of-the-box processor created the GeoEvent Definitions; the xxxx identifies the GeoEvent Extension release.

Sometimes a processor will modify an event’s structure or schema by either adding or removing a field. When a processor does this, it will create a new GeoEvent Definition. The Field Calculator Processor is a good example; you can configure this processor to place its calculated value into a new field. The Field Enricher Processor is another example; it is taking data from a feature service's table and adding new fields to the event it is processing.

When a processor creates a GeoEvent Definition for you, you are typically required to enter the name the processor should use for the GeoEvent Definition it will create.

It can be important to understand when a processor creates what I have referred to as a managed GeoEvent Definition. A processor will not create a managed GeoEvent Definition until it actually receives a GeoEvent. You have probably discovered that you cannot, for example, configure a Field Mapper Processor which follows a Field Enricher until the Field Enricher has received and processed an event. Once the Field Enricher receives a GeoEvent, it will process the event and create the necessary GeoEvent Definition. Only then can you edit your GeoEvent Service to configure your Field Mapper.

Since processors own the GeoEvent Definitions they create, any change you make to the GeoEvent Service which incorporates the processor will trigger the processor to delete its managed GeoEvent Definition when you publish the GeoEvent Service to save your changes. For this reason, you should be aware of when GeoEvent Definitions are created, when they are deleted, and when they are available for reference by another processor or output connector.

A few recommendations to take away from this quick discussion:

1. Every GeoEvent has an associated GeoEvent Definition. You should become familiar with when GeoEvent Definitions are created, which components create them, and why.
   
2. Don’t leave an input connector configured to create GeoEvent Definitions for you. When configuring an input connector to create a GeoEvent Definition, it is a recommended best practice to copy the GeoEvent Definition generated by the input connector’s adapter and then reconfigure your input to use the copy of the GeoEvent Definition.
   
3. Processors can create GeoEvent Definitions. The GeoEvent Definition associated with an event received by a processor is not necessarily going to be the GeoEvent Definition associated with the GeoEvent which comes out of a processor.
   
4. Always use a Field Mapper Processor to prepare a GeoEvent’s schema to match a feature service’s feature layer when adding/updating features. You might think you understand the structure of an event you are sending to an output – but it is a best practice to use a Field Mapper and make the schema mapping explicit.
   
5. Processors may delete GeoEvent Definitions they created. If you configured a Field Mapper “downstream” from a Field Calculator or Field Enricher, for example, the “upstream” processor may delete a GeoEvent Definition referenced by the Field Mapper. As long as you do not double-click to edit the Field Mapper’s configuration you can trust the processor “upstream” will re-create the GeoEvent Definition referenced by the Field Mapper as event data is received by the processor, before GeoEvents are received by the Field Mapper.
   
6. When publishing a stream service, never refer to a GeoEvent Definition which has been generated for you by an input connector’s adapter or a GeoEvent Definition created for you by a processor. Make a copy of any such GeoEvent Definition(s) and publish the stream service to reference a GeoEvent Definition which you own.

GeoRSS is a standard way of tagging an RSS feed so that applications can use embedded location information in each post. Using the GeoEvent Extension for ArcGIS Server, you can monitor a GeoRSS feed in real time and use it to update the applications and common operational pictures used by your colleagues. Should you encounter a secured GeoRSS feed that you would like to use, there is no standard connector that allows you to pass credentials. However, it is possible to configure a connector (without programming) that will allow you to access a GeoRSS service secured with basic HTTP authentication.

You can use the GeoEvent Manager to combine out-of-the-box transports and adapters to configure a custom connector without resorting to the GeoEvent SDK or developing any custom code.

An excellent example is available on the Support Services Blog:

• Configure Your GeoEvent 10.3 Connector to Use a Secure GeoRSS Feed

- RJ

• Debug Techniques - Configuring the application logger
• Debug Techniques - Add/Update Feature Outputs
• Debug Techniques - Application logging tips and tricks
• Debug Techniques - Geofence synchronization deep dive

How to debug the Add a Feature and Update a Feature Output Connectors is probably the question I have been asked the most over the last couple of years working on GeoEvent Extension development team – so it’s appropriate that my inaugural blog to GeoNet address it.

The scenario:

• An input appears to be successfully receiving and adapting the data from the stream and creating GeoEvents.
• The Filters and/or processors incorporated in a GeoEvent Service are handling the GeoEvents as expected.
• The event count on an Add a Feature or Update a Feature Output Connector is incrementing, but no features are being added or updated in the targeted feature layer.

So, how do you start debugging to determine what the problem might be?

My advice is to enable DEBUG logging on the feature service outbound transport to see if we can capture the JSON request being sent to ArcGIS Server and the response GeoEvent Extension receives from ArcGIS Server.

I’ve attached an image below (FeatureServiceUpdate.png) of a karaf.log I created a while ago which shows the transactions taking place when a Output Connector performs an HTTP/POST to update features in a feature service. Don’t be concerned that the illustration identifies the 10.2.2 version – the concepts and workflows are the same for the GeoEvent 10.3 and 10.4 product releases when using a traditional RDBMS.

To enable DEBUG logging on a single component in GeoEvent Manager:

• Navigate to the Logs page and click the Settings button.
• Enter the logging component in the text field Logger and select the DEBUG log level.
• Click Save.

In this case you want to log DEBUG messages for the com.esri.ges.transport.featureService.FeatureServiceOutboundTransport component only. Setting a logging level of DEBUG on the ROOT component is not recommended. Doing this will produce a very verbose set of log messages and can cause the Logs page in the GeoEvent Manager to 'hang' as it tries to refresh the rapidly updating logs.

With DEBUG logging enabled for the specified component, the GeoEvent Extension will produce more detailed logs when the feature service outbound transport handles event data. The DEBUG logging statements will include the JSON being sent and the ArcGIS Server’s response. I prefer looking at the log in a text editor, rather than using the log manager in GeoEvent Manager. You can find the karaf.log in the default folder C:\Program Files\ArcGIS\Server\GeoEventProcessor\data\log.

In the FeatureServiceUpdate.png that is attached, find the two messages time stamped 2014-03-28 14:06:09,074. The “querying for missing track id XXXX” messages indicate the GeoEvent Extension has discovered that it has not cached any information on features with the TRACK_ID “SWA1568” or “SWA510”. Looking at the third message in the series, it shows the SQL where clause used to query the …/FeatureServer/0/query REST endpoint to discover the necessary OBJECTID values.

The response from the ArcGIS Server includes a JSON array features[ ] with the geometry, OBJECTID, and unique identifier field (flightNumber in this example) for the features with the flight identifiers “SWA1568” and “SWA510”.  Notice that it took 175 milliseconds for the GeoEvent Extension to receive the ArcGIS Server response:

• 14:06:09,250 - 14:06:09,75  =  75 ms

You might find this query/response latency information valuable when profiling / debugging your GeoEvent Services which are adding or updating features through a feature service.

Once the GeoEvent Extension has the necessary OBJECTID values for the features it wants to update, it posts a block of JSON to the …/FeatureServer/0/updateFeatures REST endpoint. ArcGIS Server responds with “success”:true 325 milliseconds later:
(577 - 252  =  325).

If you find too many JSON event records being included in the transactions, making the log file difficult to read, you can try configuring your Update a Feature Output Connector to specify that the ‘Maximum Features Per Transaction’ should be limited to 1 (the default is 500). This obviously not something you would do in a production environment, but while debugging it can make the log file much easier to read.

If you find that the log is rolling over too frequently, you can edit settings in the following configuration file to allow the karaf.log to grow larger than 1MB and to keep more than 10 archival log files:

• ...\Program Files\ArcGIS\Server\GeoEvent\etc\org.ops4j.pax.logging.cfg

(Click the thumbnail above to open a larger view)

The logging package changed with the 10.6.0 release to use version 2.x of Log4J.
Settings applicable for the log4j2.appender are illustrated in the attached ops4j.pax.logging.cfg.png file.

You can also edit the message formatting specified by the layout.ConversionPattern in this configuration file to reformat the messages being written to the karaf.log - more information on that can be found here: http://www.codejava.net/coding/common-conversion-patterns-for-log4js-patternlayout

Hope this information helps –

RJ

A number of people asked me whether it is possible to use GeoEvent to do proximity alerts or collision avoidance. For example testing whether two airplanes or ships are within a certain distance between each other current positions.

The bad news is there is no Proximity operator in the current GeoEvent 10.3 to help you. The good news you can compose a solution that works even if there are thousands of tracks (of moving objects).

GeoEvent 10.3 added a number of key elements that allow composing a solution to perform this kind of operation. First is a fast in-memory spatial indexing which is a Quadtree-based to help finding an object in space very quickly. Second is Stream Service which is a WebSocket-based that allows push-based real-time streaming output. Third is the GeoFence Synchronization with Stream Service. And the fourth one is the Buffer Creator Processor. Other components are also needed.

First the input data should be output to a Stream Service as is (i.e. point geometry) to make it available for GeoFence Synchronization then a Synchronize GeoFences needs to be created by connecting to the Stream Service URL outputting from the first step. Once this is done, proximity test processing logic can be set up by first using the Buffer Creator to create a circular geometry for each point. Use GeoTagger to tag the incoming event (the results from the Buffer Creator) with the name of the GeoFence using a Spatial Operator, e.g. "Contains Any". This is pretty much a point-in-polygon test. Please note that there can be more than one GeoFence contained in the resulting field value due to the fact that a buffer polygon can contain more than one point -- which is what we want anyway. If the resulting GeoTag field is not null then trigger the proximity alert by using a filter. The buffers that were GeoTagged can be output to a feature or stream service for displaying on a WebMap in real-time. I tested this with the flight tracking data and obtained results display in a WebMap as shown below:

The March 16, 2015 Sierra Wireless Connector update is more stable and optimized. All the different RAP message types (RAP0x11, 0x12, 0x13, 0x14) are now consolidated into one and have become a field in the RAP GeoEvent definition. Several more fields and event types (e.g. Stationary (0x2C -- Stationary and 0x2D -- Started moving), Maximum Speed (0x2E -- Maximum Speed Exceeded and 0x2F -- Vehicle went back below max speed)) also added including the RAP device source IP. Odometer value parsing was also corrected. This update also supports the "Store and Forward" behavior if it is enabled on the RAP device. Store and Forward causes GPS reports to be queued up if the RAP device goes out of cellular coverage. Once the RAP device is back in cellular coverage, the GPS reports are sent to the GeoEvent Sierra Wireless input thus preventing the gap in the vehicle tracking data.

The Verizon Networkfleet connector for GeoEvent 10.3 has been updated 3/25/2015. This update added more fields to the Diagnostic message. The following fields were added:

"EventTime"

"EventTimeUTF"

"Odometer"

"EngineHours"

"TotalGallons"

"BufferType"

Here's the download location:

http://www.arcgis.com/home/item.html?id=720c28f0ef72426792012d10c37a251a

Temporal aspect can be added to a filter by using string templates.

Steps:

- Create a Filter

- Set up condition(s) comparing the $RECEIVED_TIME with values based on temporal templates

- Time period can be created by AND condition

- Multiple time periods can be created by AND and OR operator

- Exception can be done using NOT operator

For example the expression below shows to time periods that allow the asdi-FAA data to get through:

OR (Any of these are true)

     AND (All of these are true)

           $RECEIVED_TIME >= 15h 37m

           $RECEIVED_TIME < 15h 38m

     AND (All of these are true)

          $RECEIVED_TIME >= 15h 40m

          $RECEIVED_TIME < 15h 41m

Here's a sample list of temporal template:

- 2013y -- year = 2013

- 2013y 1M -- 2013/Jan

- 2013y 2M -- 2013/Feb

- 2013y 2M 25d -- 2013/Feb/25

- 2012y,2013y -- list of years for using with the IN operator

- 2013y 2M,2013y 1M,2013y 10M,2011y 2M -- list of year and months for using with the IN operator

- 2013y 2M 25d,2013y 2M 24d,2D -- list of year/month/date and year/month/day of week for using with the IN operator

- 11h -- 11 o'clock

- 12h -- 12 o'clock

- 12h 9m -- 12:09

- 12h 10m -- 12:10

- 12h 10m 10s -- 12:10:10

- 11h,12h,10h -- list of hours for using with the IN operator

- 12h 10m, 21h 5m -- list of hour/minute for using with the IN operator

- 12h 10m 11s,12h 11m 12s -- list of hour/minute/second for using with the IN operator

- 2D,4D -- list of days of week for using with the IN operator

- 2013y 2M 25d,12h 10m 10s,2D -- list of year/month/date, hour/minute/second, day of week for using with the IN operator

Note: One can use the Field Calculator to add the current time to a new field by entering currentTime() into the expression input box.

Thanks to all who attended the first Real-Time GIS special interest group (SIG) meeting at UC 2014!! Many great organizations were represented and contributed to the engaging discussions including:

• AccuWeather
• City of Quebec
• Devon Energy
• exactEarth
• FedEx Ground
• OnStar
• OSIsoft
• Pennsylvania State Police
• Tennessee Valley Authority
• Twitter
• UCSD
• USC
• White House Utility District

These organizations represent many key domains/disciplines related to real-time GIS. Several great use-cases, interesting experiences, practices, and challenges were actively discussed during the meeting. We see a lot of exciting opportunities surrounding real-time GIS and want the Real-Time GIS group to act as the collaboration platform for our community. The discussions, insights, requirements, and challenges posted by you will benefit the community as a whole and will help drive real-time GIS capabilities in the ArcGIS platform.

So, how can you help build this community? Tell a colleague, or anyone else in your professional network, to join the group and contribute. The more active members we have, the more everyone will benefit.

Thanks again to all who joined our first SIG meeting and we look forward to hearing from you here in the Real-Time GIS group as well at future events.