Skip navigation
All Places > GIS > Enterprise GIS > GeoEvent > Blog
1 2 3 Previous Next

GeoEvent

39 posts

When speaking with customers who want to get started with ArcGIS GeoEvent Server, I'm often asked if GeoEvent Server has an input connector for a specific data vendor or type of device. My answer is almost always that we prefer to integrate via REST and the question you should be asking is: "Does the vendor or device offer a RESTful API whose endpoints a GeoEvent Server input can be configured to query?"

Ideally, you want to be able to answer two integration questions:

  1. How is the data being sent to a GeoEvent Server input?
  2. How is the data formatted; what does the data's structure look like?

For example, an input can be configured to accept data sent to a GeoEvent Server hosted REST endpoint. That answers the first question - integration will occur via REST with the vendor sending data as an HTTP/POST request to a GeoEvent Server endpoint. The second question, how is the data formatted, is the focus of this blog.

What does a typical JSON data record look like?

Typically, when a data vendor sends event data formatted as JSON, there will be multiple event records organized within a list such as this:

{
    "items": [{
                  "id": 3201,
                  "status": "",
                  "calibrated": 1521135120000,
                  "location": {
                         "latitude": -117.125,
                         "longitude": 31.125
                  }
           },
           {
                  "id": 5416,
                  "status": "offline",
                  "calibrated": 1521638100000,
                  "location": {
                         "latitude": -113.325,
                         "longitude": 33.325
                  }
           },
           {
                  "id": 9823,
                  "status": "error",
                  "calibrated": 1522291320000,
                  "location": {
                         "latitude": -111.625,
                         "longitude": 35.625
                  }
           }
    ]
}

 

There are three elements, or objects, in the block of JSON data illustrated above. It would be natural to think of each element as an event record with its own "id", "status", and "location". Each event record also has a date/time the item was last "calibrated" (expressed as an epoch long integer in milliseconds).

 

What do we mean when we refer to a "multi-cardinal" JSON structure?

The JSON data illustrated above is multi-cardinal because the data has been organized within an array. We say the data structure is multi-cardinal because its cardinality, in a mathematical sense of the number of elements in a group, is more than one. The array is enclosed within a pair of square brackets:  "items": [ ... ]

If the array were a list of simple integers the data would look something like:  "values": [ 1, 3, 5, 7, 9 ]

The data elements in the illustration above are not simple integers. Each item is bracketed within curl-braces which is how JSON identifies an object. For GeoEvent Server, it is important that both the array have a name and that each object within the array have a homogeneous structure, meaning that every event record should, generally speaking, use a common schema or collection of name/value pairs to communicate the item's data.

What do we mean when we refer to a "hierarchical" JSON structure?

The data elements in the array are themselves hierarchical. Values associated with "id", "status", and "calibrated" are simple numeric, string, or Boolean values. The "location" value, on the other hand, is an object which encapsulates two child values -- "latitude" and "longitude". Because "location" organizes its data within a sub-structure the overall structure of each data element in the array is considered hierarchical.

It should be noted that the coordinate values within the "location" sub-structure can be used to create a point geometry, but "location" itself is not a geometry. This is evident by examining how a GeoEvent Definition is used to represent the data contained in the illustrated block of JSON.

Different ways of looking viewing this data using a GeoEvent Definition

In GeoEvent Server, if you were to configure a new Receive JSON on a REST Endpoint input, leaving the JSON Object Name property unspecified, selecting to have an GeoEvent Definition created for you, and specifying that the inbound adapter not attempt to construct a geometry from received attribute values, the GeoEvent Definition created would match the one illustrated below:

GeoEvent Definition

Notice the cardinality of "items" is specified as Many (the infinity sign simply means "more than one"). Also, when the block of JSON data illustrated above is sent to the input via HTTP/POST, the input's event count only increments by one, indicating that only one event record was received.

Also notice that, in this configuration, "items" is a Group element type. This implies that in addition to the structure being multi-cardinal, it's also organized as a group of elements, which in JSON is typically an array.

Finally, notice that the "location" is also a Group element type. The cardinality of "location", however, is One not Many. This tells you that the value is a single element, not an array of elements or values.

Accessing data values

Working with the structure specified in the GeoEvent Definition illustrated above, if you wanted to access the coordinate values for "latitude" or "longitude" you would have to specify which latitude and longitude you wanted. Remember, the data was received as a single event record and "items" is a list or array of elements. Each element in the array has its own set of coordinate values. Consider the following expressions:

  items[2].location.longitude

  items[2].location.latitude

The expressions above specify that the third element in the "items" list is the one in which you are interested. You cannot refer to items.location.latitude because you have not specified an index to select one of the three elements in the "items" array. The array's index is zero-based, which means the first item is at index 0, the second is at index 1, and so on.

Ingesting this data as a single event record is probably not what you would want to do. It is unlikely that an arbitrary choice to use the third element's coordinates, rather than the first or second element in the list, would appropriately represent the items in the list. These three items have significantly different geographic locations, so we should find a way to ingest them as three separate event records.

Re-configuring the data ingest

When I first mentioned configuring a Receive JSON on a REST Endpoint input to allow the illustrated block of JSON to be ingested into GeoEvent Server for processing, I indicated that the JSON Object Name property should be left unspecified. This was done to support a discussion of the data's structure.

If the illustrated JSON data were representative of data you wanted to ingest, you should specify an explicit value for the JSON Object Name parameter when configuring the GeoEvent Server input. In this case, you would specify "items" as the root node of the data structure.

Specifying "items" as the JSON Object Name tells the input to handle the data as an array of values and to ingest each item from the array as its own event record. If you make this change to our input, and delete the GeoEvent Definition it created the last time the JSON data was received, you will get a slightly different GeoEvent Definition generated as illustrated below:

 GeoEvent Definition

The first thing you should notice, when the illustrated block of JSON data is sent to the input, is the input's event count increments by three -- indicating that three event records were received by GeoEvent Server. Looking at the new GeoEvent Definition, notice there is no attribute named "items" -- the elements in the array have been split out so that the event records could be ingested separately. Also notice the cardinality of each of the event record attributes is now One. There are no lists or arrays of multiple elements in the structure specified by this GeoEvent Definition. The "location" is still a Group which is fine; each event record should have (one) location and the coordinate values can legitimately be organized as children within a sub-structure.

The updates to the structure specified in the GeoEvent Definition change how the coordinate values are accessed. Now that the event records have been separated, you can access each record's attributes without specifying one of several element indices to select an element from a list.

You should now be ready to re-configure the input to construct a geometry as well as make some minor updates to the data types of each attribute in the GeoEvent Definition in order to handle "id" as a Long and "calibrated" as a Date. You also need to add a new field of type Geometry to the GeoEvent Definition to hold the geometry being constructed.

GeoEvent Input

GeoEvent Definition

Hopefully this blog provided some additional insight on working with hierarchical and multi-cardinal JSON data structures in GeoEvent Server. If you have ideas for future blog posts, let me know, the team is always looking for ways to make you more successful with the Real-Time & Big Data GIS capabilities of ArcGIS.

When a GeoEvent Service processes an event record, the processing is generally atomic. In other words, a filter or processor considering an event record's attributes and geometry has no information on other event records previously processed and will not cache or save the current event record's attributes or geometry for later consideration by an event record not yet received.

 

There are a few exceptions - monitor processors such as the Incident Detector or Track Gap Detector necessarily cache some information in order to monitor ongoing conditions. And filters configured with ENTER or EXIT criteria need to know something about the position of the last reported event with a given TRACK_ID.

 

So how do you configure real-time analytics to compare an event's geometry against some other geometry?  You use geofences. Christopher Dufault has collected some best practices for importing, synchronizing, and using geofences in GeoEvent Server. Check out his blog Geofence Best Practices and comment with tips and tricks with geofences you've found useful in analytics you've designed.

 

- RJ

This article is the second of two articles examining enhancements made to the HTTP transport for the GeoEvent Server 10.5 release. This article examines the outbound transport. The previous article examining the inbound transport can be found here.

 

In this article, I would like to provide detail for an enhancement made to the HTTP outbound transport for the GeoEvent Server 10.5 release. The following capability is listed on the What's new in ArcGIS GeoEvent Server web help page:

  • HTTP outbound transport now supports field value substitutions in the HTTP GET mode

 

Beginning with the 10.5 product release, an output leveraging the HTTP transport can be configured to substitute event attribute values into the URL of a request GeoEvent Server will send to an external server. The attribute values are incorporated as query parameters (as opposed to the request’s content body).

The new capabilities of the HTTP transport will be described below with exercise steps you can follow to demonstrate the capabilities.

~~~

When you want to send data from event records to an external server or application you typically configure an outbound connector – such as the Push JSON to an External Website output. GeoEvent Server will incorporate the event data into the content body of a REST request and send the request to the external server as an HTTP/POST. This capability has been available in the last several releases.

A device on the edge of the Internet of Things, however, might prefer to receive requests with event data organized as query parameters rather than in a request's content body. This way the entire data payload is in the URL of the request -- leaving the content body of the request empty.

It might seem a little odd for a GeoEvent Server output, which is not intended to receive or process any type response, to make an HTTP/GET request. But the capability was introduced to enable GeoEvent Server to issue activation requests to devices which require data values be sent using query parameters.

~~~

Exercise 2A – Use HTTP/GET to send event data as query parameters to an external server

 

Why exactly are we configuring a custom outbound connector?

How's it different than the the Push JSON to an External Website connector available out-of-the-box?

 

For this exercise:

  1. Configure the following GeoEvent Server output connector.
    Browse to Site > GeoEvent > Connectors and select to create a new outbound connector. Default values for the "Shown", "Advanced", and "Hidden" properties are included beneath the illustration.


     

    Shown PropertiesDefault Value
    URL[ no default value defined ]

     

    Advanced PropertiesDefault Value
    Use URL ProxyFalse
    URL Proxy[ no default value defined ]
    HTTP Timeout (in seconds)30

     

    Hidden PropertiesDefault Value
    Formatted JSONFalse
    MIME Typetext/plain
    Acceptable MIME Typestext/plain
    Post/Put body MIME Typetext/plain
    Parameters[ no default value defined ]
    Header Parameter Name:Value List( blank )
    HTTP MethodGet
    ModeClient
  2. Save your newly configured custom outbound connector.
  3. Navigate to Services > Outputs and select to create a new (Custom) HTTP/GET request with event data as query parameters output. Configure the output as illustrated below, replacing yourServer and yourDomain with a valid server and domain for your organization.


    Note the URL specified in the illustration:

    https ://yourServer.yourDomain/server/rest/services/SampleWorldCities/MapServer/0/query?where=city_name='${Origin}'&f=json

    The format of the URL assumes that an ArcGIS web adapter (named 'server') has been configured and that an external server or client application receiving this URL could use it to query the "Sample World Cities" map service on your ArcGIS Server. GeoEvent Server will substitute the variable ${Origin} in the URL's query parameter with an actual attribute value from a received event record, enabling the external server or client application to make a more specific query based on real-time events.
     
  4. Save your updated output, then publish a GeoEvent Service which incorporates your output and an input of your choice. You can use any type of input, so long as the GeoEvent Definition associated with event records received by the input includes an attribute field named Origin.

    Queries through a web adapter to a Portal secured web service from an unauthenticated source will return an error. Since the Sample World Cities web service is secured by Portal in my current deployment, I expect the request made by GeoEvent Server will generate an error. In order to complete the demonstration we will use the GeoEvent Server's debug logs to confirm that the output has constructed a valid query and sent the request to the ArcGIS Server map service.
  5. Navigate to the Logs page in GeoEvent Manager. Click 'Settings' and enable DEBUG logging for the feature service outbound transport logger (com.esri.ges.transport.http.HttpOutboundTransport).
  6. Send an event record to your GeoEvent Server input whose Origin attribute is the name of one of the cities in the Sample World Cities map service (e.g. Chicago). Refresh the Logs page in GeoEvent Manager and you should see log messages with information similar to the following:

 

The first message shows that 'Chicago' was indeed substituted into the query parameters by the GeoEvent Server output and a request was made. The error may or may not be displayed; as indicated above, the map service in my case is Portal secured and this request did not include a token authenticating the request.

 

There are a couple of things you'll want to keep in mind. The URL you use to configure the the output must URL Encode its query parameters to make them HTTP safe. But the value is being substituted by GeoEvent Server is based on a string received from a real-time data source. This means you may have some work to do to make sure that "San Francisco" is represented as San%20Francisco not San Francisco before an event record is sent to an output.

 

Also, the enhancement being introduced in this article was designed specifically for HTTP/GET since those requests do not include a JSON payload in the request’s body. However, some rudimentary testing suggests that you can use HTTP/POST as well; I suppose it would be up to the external server receiving the request whether or not to honor an HTTP/POST and either ignore the request’s JSON payload or potentially consider its content in addition to the values in the query parameter.

 

Finally, you do have some freedom in how the request’s query string is specified. For example, you could construct a parameterized string something like; GeoEvent Server will handle the substitution of the multiple parameter values:

query?where=city_name+IN+%28%27${CityA}%27%2C%27${CityB}%27%29&f=json

 

If you send the string highlighted above through an HTML decoder you'll see that it is equivalent to:

where=city_name IN ('${CityA}','${CityB}')&f=json

 

I hope these two blogs were helpful.  Please comment below with questions and I'll do my best to answer them.

 

-- RJ

This article is the first of two articles examining enhancements made to the HTTP transport for the GeoEvent Server 10.5 release. This article examines the inbound transport. The second article examining the outbound transport can be found here.

 

In this article, I would like to provide detail for an enhancement made to the HTTP inbound transport for the GeoEvent Server 10.5 release. The following capability is listed on the What's new in ArcGIS GeoEvent Server web help page:

  • HTTP inbound transport now accepts GET requests in the query parameters

 

Beginning with the 10.5 product release, an input leveraging the HTTP transport can be configured to support an external server or application which incorporates its data payload in the URL of the request (as opposed to the request’s content body).

The new capabilities of the HTTP transport will be described below with exercise steps you can follow to demonstrate the capabilities.

~~~

When you want to receive event records as an HTTP/POST request from an external server or application you typically configure an inbound connector – such as the Receive JSON on a REST Endpoint input. GeoEvent Server will create a REST endpoint to which the external server can post its event data with the event data included in the content body of the request. This capability has been available in the last several releases.

A device on the edge of the Internet of Things, however, might prefer to organize the event data as query parameters and incorporate its data payload in the URL of the request -- leaving the content body of the request empty. For example:

  • http :// localhost:6080/geoevent/rest/receiver/http-receiver?field1=v1&field2=v2&field3=v3
  • http :// localhost:6080/geoevent/rest/receiver/http-receiver?data=v1,v2,v3

Beginning with the 10.5 product release an input pairing either the out-of-the-box JSON or TEXT adapter with the HTTP inbound transport can be configured to support the use cases above with an HTTP/GET request.

~~~

Exercise 1A – Use HTTP/GET requests to send event data to GeoEvent Server as query parameters

  1. Create the following GeoEvent Definition

  2. Configure the following GeoEvent Server input connector


    Note the new 10.5 parameter:  Get Request Contains Raw Data

    Review the help tip provided for this parameter. If the inbound connector is running in SERVER mode and receives an HTTP/GET request, if the request content body is empty and the request URL includes query parameters, the default (‘No’) will consider each name/value pair as a separate attribute value in an event record. If the default is changed to ‘Yes’ you will be expected to specify the one query parameter which will be considered the event’s raw data.

  3. Configure a GeoEvent Server output connector and publish a GeoEvent Service

    You can use any outbound connector which supports JSON event record displays. Recommended output connectors are ‘Send Features to a Stream Service’ or ‘Write to a JSON File’.




  4. Send the following HTTP/GET request to your input connector’s endpoint

    http://yourServer.yourDomain:6180/geoevent/rest/receiver/rest-json-in?fname=Robert&lname=Lawrenson&employee_id=123

 

You should observe the event count of your ‘Receive JSON on a REST Endpoint’ input increment as HTTP/GET requests are made on your input’s REST endpoint

~~~

Exercise 1B – Explore HTTP/GET requests whose query parameters include comma delimited values

Rather than incorporating the event data into a series of key/value pairs, the event data can be conveyed using a single query parameter whose value is a set of comma delimited values. The delimited text values will require an inbound connector which leverages the TEXT adapter (rather than the JSON adapter used in the previous exercise).

GeoEvent Server does not include a “Receive TEXT on a REST Endpoint” inbound connector out-of-the-box, so you will need to configure one for this exercise.

  1. Configure the following GeoEvent Server input connector.
    Browse to Site > GeoEvent > Connectors and select to create a new inbound connector. Default values for the "Shown", "Advanced", and "Hidden" properties are included beneath the illustration.



    Shown PropertiesDefault Value
    Event Separator\n (newline)
    Field Separator, (comma)
    Incoming Data Contains GeoEvent DefinitionFalse
    Create Unrecognized Event DefinitionsFalse
    Create Fixed GeoEvent DefinitionsFalse
    GeoEvent Definition Name (New)[ no default value defined ]
    GeoEvent Definition Name (Existing)[ no default value defined ]
    Language for Number Formatting[ no default value defined ]

    Advanced PropertiesDefault Value
    Acceptable MIME Types (Server Mode)text/plain
    Expected Date Format[ no default value defined ]
    Build Geometry From FieldsFalse
    X Geometry Field[ no default value defined ]
    Y Geometry Field[ no default value defined ]
    Z Geometry Field[ no default value defined ]
    Well Known Text Geometry Field[ no default value defined ]
    wkid Geometry Field[ no default value defined ]
    Get Request Contains Raw DataTrue
    Parameter Name for the Raw Datadata

    Hidden PropertiesDefault Value
    ModeServer
    Use Long PollingFalse
    Frequency (in seconds)[ no default value defined ]
    Receive New Data OnlyFalse
    Post/Put body MIME Type[ no default value defined ]
    HTTP MethodGet
    Header Parameter Name:Value List( blank )
    Post/Put FromParameters
    Post/Put Parameters( blank )
    Content Body[ no default value defined ]
    Parameters[ no default value defined ]
    URL[ no default value defined ]
    URL Proxy[ no default value defined ]
    Use URL ProxyFalse
    Acceptable MIME Types (Client Mode)[ no default value defined ]
    HTTP Timeout (in seconds)30
    Append to the End of Payload[ no default value defined ]
  2. Save your newly configured custom inbound connector.
  3. Navigate to Services > Inputs and select to create a new (Custom) Receive TEXT on a REST Endpoint input.
    Configure the input as illustrated below. Use the GeoEvent Definition you created for the last exercise.


  4. Publish a GeoEvent Service which incorporates your newly configured input and any outbound connector which supports JSON event record displays. You can use the outputs configured for the previous exercise if you wish.
  5. Send the following HTTP/GET request to your input connector’s endpoint (note the endpoint's name has changed):

    http://yourServer.yourDomain:6180/geoevent/rest/receiver/custom-receive-text-rest-in?data=Robert,Lawrenson,123

 

You should observe the event count of your ‘(Custom) Receive TEXT on a REST Endpoint’ input increment as HTTP/GET requests are made on your input’s REST endpoint.

~~~

On both the Linux and Windows platforms, GeoEvent Server is run from within a Java Virtual Machine (JVM) instance. The out-of-the-box default configuration allocates only 4GB of your server's available RAM to this JVM. All GeoEvent Server operations requiring RAM draw from this allocation.

 

Some reasons you might want to increase the amount of RAM allocated to the GeoEvent Server's JVM include:

  • A need to load a large number of geofences into the GeoEvent Server's geofence manager
  • A need to process a large velocity or large volume of event records (more than a few hundred per second)
  • A need to cache a large amount of information from a secondary enrichment source for event record enrichment
  • An expectation that real-time analytics using Incident Detectors will generate a large number of concurrent incidents
  • An expectation that real-time analytics requiring state (e.g. Track Gap detection and monitoring, or spatial conditions such as ENTER / EXIT) will need to work with a large number of assets with unique track identifiers

 

System administrators who have determined that their server machine has sufficient available RAM, and who have also determined that their GeoEvent Server deployment has a need to allocate more RAM to the JVM instance running GeoEvent Server, can follow the steps outlined below to increase the memory available to GeoEvent Server by allocating more RAM to the hosting JVM.

 

  1. Stop GeoEvent Server
    • On a Windows platform, make sure the GeoEvent Server Windows Service and its associated java.exe process are stopped.
  2. Open the ArcGISGeoEvent.cfg configuration file in a text editor
    • On a Windows platform, this file is found in the ...\ArcGIS\Server\GeoEvent\etc folder by default
    • When located beneath C:\Program Files you will need to edit this file as a user with administrative privilege
  3. Locate the block of JVM Parameters in the file
    • Note that at different releases the indexes for the JVM parameters will be different from the illustration below
    • Click the image below for an enlarged view in a new tab / window:

  4. Increase the -Xmx parameter for the Java Heap Size from its default (4096m) to specify a larger allocation
    • For example:   -Xmx8192m
    • Note that the allocation is in megabytes
  5. Save your edits to the ArcGISGeoEvent.cfg file (and dismiss your text editor)
  6. Start GeoEvent Server

 

Using system administrative tools you should be able to verify that the JVM instance (java.exe process) never consumes more memory than what is allocated by the ArcGISGeoEvent.cfg configuration file, and that more than the default 4GB is now available for GeoEvent Server operations.

Hello Everyone --

 

I've recently completed three short videos which illustrate how to use stream services and capabilities related to stream services -- specifically 'Store Latest' and 'Related Features' which were never covered in the product tutorial available on line.

 

We are working on updating the tutorial's exercises and narrative to be consistent with these new videos, but I don't want to hold the videos until the tutorial re-write is complete. (The videos will eventually be bundled with the tutorial for download.)

 

The basic stream service capability provided by GeoEvent did not change with the ArcGIS 10.5 product release. However, some minor changes in behavior were made with regard to 'Store Latest' when working within different enterprise configurations, such as single-machine vs. multi-machine and when you have federated with a Portal for ArcGIS vs. when you have not federated with a Portal.

 

Enhancements to the 'Related Features' configuration workflow now allow you to select the feature service from which related features will be obtained (rather than having to manually enter the URL of an existing feature service).

 

Three MP4 files have been attached to this blog.  Please check-out the videos -- they are each only 10 to 15 minutes. Let the team know (e-mail geoevent@esri.com) if you think bundling a short video with a less detailed tutorial is an approach which works for introducing product updates and documenting product functionality.

 

Best Regards --

RJ

A couple of times a year a script developer will ask me about using the GeoEvent Admin API to automate some administrative task - such as stopping and restarting a GeoEvent input.

Any user action taken through the GeoEvent Manager web application makes a request against a URL in our GeoEvent Admin API. So, in theory, once you authenticate with the GeoEvent Admin API, you should be able to script some fairly simple tasks, like stopping a running input, modifying one of the input's parameters, saving the input's new configuration and restarting the input to begin receiving event data.

I'd like to share a blog post by Andy Ommen, a solution engineer working with Esri Database Services out of our Boston regional office. Take a look and let him know if you find his information useful. I really appreciate him sharing this out through his blog.  Here's the link:  Scripting tasks using the GeoEvent Admin API

Update March 2019Eric Ironside, a product engineer on the Real-Time team, has created a second blog illustrating how to update the properties of a GeoEvent Input. Much appreciated Eric!

Update November 2019 – Jake Skinner, a platform configuration engineer in Esri's Philadelphia region, has another blog illustrating how to authenticate with the GeoEvent Server administrative API in order to script administrative actions like updating the configurable properties of inputs and outputs. Thanks Jake!

It might also be helpful to know that the ArcGIS Server Administrative REST API is documented using Swagger. You can review available operations exposed by the Admin API:

  1. Browse to https://my-machine.domain:6143/geoevent/admin
  2. Acquire a token from your ArcGIS Server (or Portal for ArcGIS if federated) and log-in
  3. In the top-left corner, click the API link to take you to the Swagger Doc for the GeoEvent Server Admin API
  4. Note the advice at the top of the page on how to authenticate your admin script's requests with the API

Hope this information is helpful –
RJ

When processing event records which include a large number of unique track identifiers you might notice that some spatial relationships evaluated by GeoEvent filters and processors do not behave as expected. We will focus on the “Enter Any” and “Exit Any” spatial operators as they apply to a GeoTagger Processor when more than 1000 unique TRACK_ID values are present. I will explain in detail the behavior reported to me by a few users and a potential product configuration you can make to better accommodate your data.

 

Consider, for example, a GeoTagger Processor configured to enrich event records with the name of a geofence. The processor will evaluate a set of geofences and add a new field with the name of a geofence to an event record whenever the event’s geometry enters or exits an area. As you observe the output from a GeoEvent Service however, you notice that events are being dropped at random from the processor. Events that you observed several minutes ago are being removed or are severely delayed in displaying at all within the GeoEvent Service and are not included in the output. While there may be other reasons for your observations, we’ll assume that the GeoTagger Processor is the root cause.

GeoTagger Properties

 

"Enter Any" and "Exit Any" Spatial Operators maintain state

The majority of spatial operators do not require GeoEvent to maintain state information. The “Enter Any” and “Exit Any” operators are the exception, and require GeoEvent to track both geometry and track identifier as events are moved forward through the processor. Maintaining state requires a prior knowledge of each event’s location; each event is treated as a dependent of its previously observed location. Without maintaining state however, the previous positions of the events remain unknown to GeoEvent and all events are treated independently. This means that a GeoTagger Processor configured with the “Enter Any” or “Exit Any” operation must utilize a cache of unique track identifiers for every observed event. In short, this is what is known as a cache‑aware processor node.

 

GeoEvent does have a maximum cache size enabled which is in place to maintain the state of all events as efficiently as possible; the default value for this property is 1000 events. When an event arrives at a cache-aware node and the event’s TRACK_ID is not contained in the cache, one of the previously observed TRACK_ID values must be discarded if the cache is currently full. This is done to make room for the newly observed event and respect the 1000 event limit.

What does this mean for your data? Conceptually it means that the processor will forget that it has ever encountered an event with the discarded TRACK_ID. When observing the behavior in real-time, events will spontaneously be dropped from the processor and certain events that you observed several minutes ago may not be displaying at all in the output destination.

 

Though the logic remains the same for both, the definitions used for “Enter Any” and “Exit Any” are in fact different. In order for GeoEvent to recognize that an event’s geometry has entered a geofence, its prior location must have been observed outside that geofence. Conversely, GeoEvent will not recognize that an event’s geometry has exited a geofence unless the geometry’s prior location was observed inside that geofence. These definitions are honored by default, but can produce different results if an additional property that GeoEvent uses to evaluate the spatial relationships is changed.

 

"First GeoEvent triggers Enter" and "First GeoEvent triggers Exit"

When a cache-aware node must make an “enter or exit” decision, it respects the settings of two properties within the GeoEvent Manager. These are the "First GeoEvent triggers Enter" and "First GeoEvent triggers Exit" properties, which determine the importance of the “enter” or “exit” operations. By default GeoEvent assumes that “entry” is more important than “exit” and that most event geometries are already outside of a geofence. The defaults for these properties are true and false respectively. You can change the values for each, but it is recommended that the change be made only with deliberate care. If not careful you could easily configure GeoEvent to start generating unwanted analysis and notifications, particularly after a restart of the server machine. For example, even if event geometries are expected to move around inside a geofence such as an administrative boundary, they will be located outside every other geofence that is registered with GeoEvent. If you change the "First GeoEvent triggers Exit" property from false to true you may unexpectedly get a significant number of “exit” evaluations every time the server machine is rebooted.

 

We’ll look at an example of a default configuration with GeoEvent using “entry” as the main importance. Let’s assume that a GeoTagger is set to an “Enter Any” operation and the event TRACK_ID is not in the cache. If the event’s geometry lies inside of a geofence, then the GeoTagger will read the event as having entered. Alternately, if this same GeoTagger is set to “Exit Any” and the event’s geometry is already outside of a geofence, the processor will determine that the point did not exit. Sound tricky? Just remember that GeoEvent assumes that geofences are empty 99% of the time, and that points are expected to enter at some future point. That doesn’t mean that exits are ignored, but they are placed with less importance and require more observations to determine.

 

Changing the maximum cache size for cache‑aware nodes

The default setting for the maximum cache size is exposed through a specific product configuration file.  It is important to note that this cache size is a maximum for each cache-aware node, and it is not a system-wide limit. You must have administrative privilege to edit this file. Changing the default value can result in your GeoEvent Server consuming significantly more RAM. The Java process in which the GeoEvent Server runs, by default, is limited to 4GB of system RAM. If every cache-aware node begins caching event data for significantly more than 1000 unique track identifiers, a larger portion of the 4GB will be consumed leaving less room for other more basic functions.

 

To promote system stability it is recommended that you estimate of the total number of unique track identifiers expected from your event data and set the maximum cache size value slightly higher than the estimate (to allow for more features to be added over time). Keep the value as small as possible and do not specify an arbitrarily high maximum cache size.

The cache value is contained within the com.esri.ges.manager.servicemanager.cfg file located in the following directory on a default system:

 

“C:\Program Files\ArcGIS\Server\GeoEvent\etc”

 

Keep in mind that this location may change if GeoEvent Server was installed to directories other than the product’s default system folder.

 

Incident Detection and the 1000 event cache limit

Within GeoEvent, you could also observe that open incidents are being dropped from the output destination in a similar manner to how the GeoTagger discards events. The Incident Detector Processor is another cache-aware node in GeoEvent, and utilizes an incident cache particularly when evaluating conditions for concurrently open incidents. Much like the GeoTagger, the Incident Detector will discard open/ongoing incidents if its incident cache is full and a new event triggers an incident to open. The opening/closing of incidents is managed by the Incident Manager program, which runs in the background to maintain state. The Incident Manager has a 1000 incident cache limit enabled by default, but it is exposed through the GeoEvent Manager rather than a configuration file. This default size can also be changed, but the same recommendations as the GeoTagger Processor apply. Obtain an estimate first of how many unique features GeoEvent will be processing, and then determine how many probable incidents could be open at one time. Set the number of Open and Closed incidents to slightly higher than the calculated maximum.

 Incident Manager Settings

The Real-Time GIS product team is thrilled to announce our newest addition to the ArcGIS GeoEvent Gallery, the Waze Connector for GeoEvent. 

 
The Waze Connector for GeoEvent allows users to receive live data from Waze, the world's largest community-based traffic and navigation app. Through the free Waze Connected Citizens Program (CCP), they support a two-way data exchange between Waze and their municipal partners. This allows for streamlined access to both authoritative information alongside user submitted alerts and hazards.

 

In order to utilize the Waze data in GeoEvent Extension you must first be a part of their Connected Citizens Program. Information regarding the requirements to participate and apply to join that program can be found at the following link:
 

Waze Connected Citizens Program

https://www.waze.com/ccp 

Once you've received your necessary access credentials you can begin pulling in the live user submitted Alerts and Traffic Jams and consuming that data throughout the ArcGIS platform. The tutorial included with the connector will walk you through the process of polling the feed, filtering by data type, and writing that data out into the spatiotemporal big data store or other data sources. Also included is an optional segment that will allow users to quickly apply the Waze symbology to their newly created web maps.

 

 

For an example of the types of information being made available to Esri users, and a sneak peak of the new aggregation styles for map services using the spatiotemporal big data store with the 10.5 release, check out the screenshots from our demo application below.

 

 

 

 

It is strongly recommended that users who want to integrate the GeoEvent Extension and Portal for ArcGIS in the same environment use the most recent release of the ArcGIS product. GeoEvent’s integration with Portal is not supported prior to the 10.4 product release. Some users have successfully deployed these products using earlier releases, however, their deployments have significant functional limitations and known constraints. GeoEvent first began integrating Portal’s updated security model at the 10.4 release; integration was completed with the 10.4.1 product release. This blog clarifies GeoEvent’s integration with Portal, provides detail on how the integration works, when and why it was implemented the way it was, and suggests some considerations you should be aware of when planning a deployment.

 

 

How Integration Works

 

When we discuss integration between GeoEvent and Portal, we're really talking about two different issues.

 

The first involves security and access to the GeoEvent Manager. As the product name implies, GeoEvent is an extension for ArcGIS Server. Beginning with the 10.3 release of both products, GeoEvent uses the same security store as ArcGIS Server. What that means in practical terms is that when logging into the GeoEvent Manager to configure inputs, outputs, and services, you do so using an administrative account for ArcGIS Server. It is recommended that you use Server’s default Primary Site Administrator (PSA) account, but any administrative account – such an Integrated Windows Authentication (IWA) admin account – can be used. Once inside the GeoEvent Manager web application the user experience is the same regardless of which administrative account was used to gain access.

 

When ArcGIS Server has been federated with Portal for ArcGIS, it gives up its own security store and relies on Portal to authenticate and authorize user access. This means that GeoEvent, in turn, will also rely on Portal’s security model since GeoEvent is using the same security model used by ArcGIS Server.

 

The second issue involves access to data. GeoEvent uses server connections, registered as GeoEvent Data Stores, to connect to server machines and request access to data. When registering a server connection a user provides the URL of the server they are trying to reach and, if necessary, credentials or a token to be used when accessing the server content. You can register server connections to the local ArcGIS Server (the one running the GeoEvent Extension), an external ArcGIS Server instance, an ArcGIS Online for Organizations instance, or an instance of Portal for ArcGIS (either on the local server or running on an external server).

 

For "FULL" Integration to be considered, both issues must be addressed. Users should be able to log into GeoEvent Manager using the Portal Security Store, AND be able to access any potential data source coming from Portal for ArcGIS.

 

When and Why Was This Implemented?

 

As indicated earlier, GeoEvent is an extension to the ArcGIS Server product. A decision was made at the 10.3 release that GeoEvent should use a token-based security model, similar to what ArcGIS Server uses, to simplify access control as well as the user login experience. This worked by passing an administrative user's credentials to ArcGIS Server and receiving back an encrypted token which verified the user's access, limited which server / site could use that token, and imposed an expiration date after which connections using the token would no longer validate. This worked well for ArcGIS Server in that GeoEvent could easily log in using those same credentials, and could use a Long Term Token to validate access to all of Server's services through the Data Store for the life of that token, up to one year.

 

When configured with built-in users, Portal's security worked the same way. Users would provide credentials at the login page for the GeoEvent Manager which GeoEvent would use to request a token from Portal on behalf of the user. When IWA had been configured this approach required users to first use Portal’s Token generation page to obtain a token (by entering credentials recognized by Portal) and then entering that token into the GeoEvent Manager’s login page.

 

This exposed a crucial difference between Server tokens and Portal tokens – specifically the maximum time a token could be used before it expired. While Server allowed tokens to be created which would not expire for a year, Portal only supported tokens with a maximum life of up to two weeks. This wasn’t a significant limitation for a user’s login to GeoEvent Manager, but it meant that GeoEvent Data Stores would need to be reconfigured with a new token every two weeks.

 

At the 10.4 release Portal updated the federation experience with ArcGIS Server, providing an opportunity for GeoEvent to improve its integration and utilize the same OAuth2 security model. Portal security integration was completed with the GeoEvent 10.4.1 product release. This removed the need to have different login workflows to authenticate built-in and IWA users, and more importantly removed restrictions inherent with the token-based access developed for the 10.3 / 10.3.1 releases of GeoEvent. Within the Data Store, users could choose to enter credentials which GeoEvent would encrypt and use to authenticate requests it made to either Server or Portal. These changes addressed both issues identified above streamlining a user’s login to GeoEvent Manager and providing long term access to Portal data and resources.

 

 

Deployment Considerations

 

GeoEvent deployment into a federated environment with Portal prior to 10.4 is not supported by Esri Technical Support. While it may be possible to architect a solution using these products at the 10.3.x releases, it is against our recommended best practices. Changes made for 10.4 and 10.4.1 to replace the reliance on token-based authentication cannot be ported back to earlier releases. There will be no patches or hot fixes to re-architect the token-based security as it would involve significant modification to Portal, Server, and GeoEvent.

 

Users looking to leverage GeoEvent and Portal in an enterprise solution are encouraged to use the latest public release of each product as these will incorporate improvements to both the user experience and product integration.

The GeoEvent product team would like to announce the release of a new tutorial introducing the Spatiotemporal Big Data Store. The tutorial is available on the GeoEvent product gallery with our other product tutorials.

 

Here is a direct link to the new tutorial:  Tutorial - Spatiotemporal Big Data store

 

Feedback is welcome. Please add any comments you have to the item in the gallery or send an e-mail to geoevent@esri.com

 

Best Regards --

RJ

Changes made to the Text adaptor for the 10.4 product release might catch a few folks off guard, so I want to call attention to what you might expect to see and why.

 

 

Several inbound connectors – specifically those using the Generic-JSON and XML adaptors – provide the ability to ‘Construct Geometry From Fields’ so that when event data is received which contains coordinate information on a point location a Geometry can be constructed by the GeoEvent input prior to sending the event to a GeoEvent Service.

 

At the 10.3.1 release, inputs such as ‘Receive Text from a TCP Socket’ and ‘Watch a Folder for New CSV Files‘ do not have this capability because it was not included in the Text adapter.

 

It was observed that if a user allowed the input to create a GeoEvent Definition and then later reconfigured the input to request that it construct a Geometry from specified event fields without also editing the event definition to include a Geometry field in which the constructed Geometry could be placed … that the input would stop ingesting event data.

 

For the 10.4 release, an enhancement was made to the Text adaptor to bring it in line with the Generic-JSON and XML adaptors:

 

Issue #945:  Text adapter needs to follow Generic-JSON adapter's behavior and create a field of type Geometry, tagged GEOMETRY, when creating a GeoEvent Definition

 

Now, at the 10.4 release, when you select the ‘Construct Geometry From Fields’ option and specify the name of event attribute fields containing coordinate values, the GeoEvent Definition generated for you by the input will include a field named “Geometry”.  The field will be added as the last field in the event definition and will be tagged with the GEOMETRY GeoTag.

 

Capture.png

Capture1.png

 

If you leave the ‘Construct Geometry From Fields’ parameter set to its default ‘No’ the generated event definition will not have the Geometry field highlighted above.

 

If you elect to edit the generated GeoEvent Definition to replace ‘Field1’, ‘Field2’, ‘Field3’ (etc.) with meaningful names, but neglect to include a field in the GeoEvent Definition into which the constructed Geometry can be placed, and then alter your input to switch ‘Construct Geometry From Fields’ from ‘No’ to ‘Yes’ … beginning with the 10.4 release the input will alter the event definition for you to create the needed Geometry field.

 

Here’s where this enhancement might produce undesired behavior …

 

Let’s say that you had been using GeoEvent at the 10.3.1 release with a ‘Receive Text from a TCP Socket’ input successfully receiving event data sent from the GeoEvent Simulator. Your GeoEvent Definition might look something like the following:

 

Capture2.png

 

You export your 10.3.1 GeoEvent product configuration as an XML file, upgrade to 10.4, and import your configuration. When you try simulating the same data which was working at the 10.3.1 release, the input’s event count no longer increments and it does not appear that your GeoEvent is receiving any of the data you’re simulating. You look in the GeoEvent logs and see a message:

 

Cannot find matching GeoEvent Definition. Event is not created.

The incoming text is SWA2706,3/16/2012 02:25:30 PM,IAD,TPA,B733,37000,-79.585739,34.265521

 

What’s going on?

 

It is likely that the field named “Location” is not the field the ‘Construct Geometry From Fields’ operation expects. Even though you were careful to copy and edit the GeoEvent Definition, back in 10.3.1, to include a field whose data type is Geometry and tag the field GEOMETRY … its name is not “Geometry”, so the input doesn’t recognize it.

 

Remember that delimited text is not well-defined like JSON or XML. If the inbound events contained field names then the adapter would be able to map the received data into the correct fields in the event definition. The 10.4 Text adapter does not assume that a field of type Geometry is sufficient (even the field is tagged GEOMETRY). It looks for a field named “Geometry” and, failing to find one, it logs the error shown above and discards the event.

 

To fix the problem, all you have to do is edit the GeoEvent Definition imported with your configuration and rename the last field from “Location” to “Geometry”.

 

This is probably a scenario that many users will encounter when upgrading 10.3.1 configurations which include a TCP/Text input to the 10.4 release. You need to remember that at 10.4 your Geometry fields should be named “Geometry” (not “Location”, or “Target”, or “Geom”). The field must still be the last field in the GeoEvent Definition.

 

Hope this information saves you a little future grief -

RJ

I received a second request recently for the information below, provided to me originally by Javier on the GeoEvent Server development team, so I thought I’d post it in case others are looking for the information.

 

See also:  GeoEvent WebSockets in 10.6 with ARR

 

 

From: Javier Delgadillo

Sent: December 01, 2015

 

Since the ArcGIS WebAdaptor does not support GeoEvent Server, you cannot use it to proxy GeoEvent requests. You can however configure Application Request Routing (ARR) and create your own rules to allow IIS to proxy the requests. Below is a screenshot of a rule created within IIS 8.5 to allow proxy of GeoEvent Server via IIS:

 

Capture.png

 

After installing ARR, double-click the URL Rewrite icon and create a similar rule (replacing the hostname of the Rewrite URL to match your environment):

Capture1.png

 

After applying the rule confirm you are able to access GeoEvent Manager through IIS:

Capture2.png

 

Configuring your SSL certificates will be important. You can use IIS to create and export a Domain Certificate with a private key and then configure ArcGIS Server and GeoEvent to use that certificate. An alternative would be to get IIS to trust the certificate that is configured with your ArcGIS Server/GeoEvent installation.

 

It was discovered that, using the 10.3.1 release of GeoEvent Server and IIS version 8.5 with WebSockets enabled, with ARR configured to create a reverse proxy for GeoEvent that StreamService connections did not work. This issue was addressed in the 10.4 release of GeoEvent Server.

 

10.3.1 deployments can create a reverse proxy similar to WebAdaptor specifically for WebSockets using NGINX.  Steps outlined below provide some detail on installing and configuring NGINX. A sample nginx.conf configuration is attached which you can use as a reference.

 

Please note:  Esri tech support cannot help troubleshoot reverse proxy configurations - whether they be NGINX, IIS, or Apache.  If you do plan on deploying a reverse proxy as part of a systems solution, make sure you have someone with appropriate experience available to you for help troubleshooting. Esri tech support has a KB article you can refer to:

http://support.esri.com/cn/knowledgebase/techarticles/detail/40680

 

  • Install a WebSocket reverse proxy server Nginx.
  • Enable HTTPS on the proxy server and configure it to use a certificate issued by a trusted 3rd party CA (Thawte, VeriSign, DigiCert).
  • Configure the proxy server to forward requests to GeoEvent Services. Attached is an example of an Nginx configuration file. You will need to change the following settings:
    • server_name <your Web Socket reverse proxy server name>
    • ssl_certificate <your CA-cert certificate file>
    • ssl_certificate_key <your CA-cert certificate key file>
    • server <list of servers that GeoEvent Extension is running on>. In this example, the server names are dev01014.esri.com, dev01015.esri.com, and dev01018.esri.com

 

Note: The WebSocket reverse proxy server is set to use HTTPS, but it connects to HTTP on the backend. In the example configuration file, the proxy_pass for port 443 is set to http: //skivmHTTP; not https: //skivmHTTP.

 

  • In a web browser, navigate to http: //<ServerName>:6080/arcgis/admin to access the ArcGIS Server Administrator Directory.
  • Login and click system -> properties-> update.
  • Enter a property called WebSocketContextURL to point to the Web Socket reverse proxy.

 

For example:

{"WebSocketContextURL": "wss://skivm.esri.com"}

 

  • In a browser, navigate to:

https: //<WAMachineName>/<context>/rest/services/<StreamServiceName>/Streamserver/subscribe

 

  • Click Subscribe to verify data is streaming. You can also verify data is streaming by navigating to:

https: //<WAMachineName>/<context>/rest/services/<StreamServicename>/StreamServer?f=jsapi

 

See Also:

https://nginx.com/resources/wiki

http://nginx.com/resources/admin-guide/reverse-proxy

http://www.iis.net/downloads/microsoft/application-request-routing

http://www.iis.net/learn/extensions/configuring-application-request-routing-(arr)

 

 

Hope this information is helpful -

RJ

On December 8, 2015, the GeoEvent team has released the update to the FlightAware connector. This is the Release 2 that adapts to the message field order changed by FlightAware. All users should adopt this update as the previous release is no longer functional.

After adding the Verizon Networkfleet input into GeoEvent Server for the first time, it is time to test it. This can be done without waiting for the actual data to come from the Verizon Networkfleet server and it is a good preparation step to ensure that the internal network behind your organization firewall is in a ready state. The UI of the Verizon Networkfleet input should look similar to the illustration below:

NetworkfleetConnector.PNG

 

Highlight the string value next to the URL and copy the string from localhost to the end of the long GUID to the clipboard (e.g. by using ctrl-c on the keyboard).

 

Using a web browser like Google Chrome or Mozilla Firefox, you can simulate data -- Chrome will be used in this example. Launch Chrome then click on the Apps button in the upper left corner.

 

GoogleChrome.PNG

 

If you already have the DHC app then it will show up on the browser. Otherwise click on the Web Store, search for the DHC app, and add it to Chrome.

 

ChromeApps.PNG

 

Click on the DHC app, Chrome will open another browser window. Fill in the DHC input as shown below:

 

ChromeDHC-Networkfleet.PNG

 

- Set the REQUEST to HTTP or HTTPS and POST

- Paste the URL copied from the Verizon Networkfleet input into the input box next to HTTP --  if the HTTP is selected then use 6180 after localhost or 6143 for HTTPS. For HTTPS the browser should accept the certificate from the GeoEvent Server instance otherwise posting will fail.

ChromeDHC-Networkfleet-https.PNG

 

- Set the Content-Type to application/x-networkcar-xml

- Copy the test XML from below into the BODY

 

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE NetworkfleetMessage SYSTEM "http://www.networkfleet.com/dtd/dataconnect/DataConnect.v2.1.dtd">

<NetworkfleetMessage Type="GPS">

<VIN>WAUG5678901234567</VIN>

<FleetId>6543</FleetId>

<MsgId>1761773890</MsgId>

<MessageTime>2007-10-03 22:42:13 GMT</MessageTime>

<MessageTimeUTF>1191451333</MessageTimeUTF>

<DeliveryStatus>Current</DeliveryStatus>

<GPSFixes NumFixes="1">

<GPSFix>

<FixTime>2015-1-15 22:42:13 GMT</FixTime>

<FixTimeUTF>1191451333</FixTimeUTF>

<Latitude>39.87545</Latitude>

<Longitude>-119.41060</Longitude>

<Ignition>On</Ignition>

<Speed Type="Avg" Units="MPH">12</Speed>

<Speed Type="Inst" Units="MPH">15</Speed>

<Speed Type="Max" Units="MPH">27</Speed>

<Heading>224</Heading>

<Odometer>94068.528</Odometer>

<AgeInMiles>0.012</AgeInMiles>

</GPSFix>

</GPSFixes>

</NetworkfleetMessage>

 

Before clicking the Send button, open another Chrome window and log in to GeoEvent Manager and navigate to the Monitor page. Click Reset Statistics to make sure all the counts are set to 0 and ensure the Verizon Networkfleet input is started. In the DHC app, click Send.

 

GEP-Monitor.PNG

NetworlfleetCounter.PNG

 

If all goes well the DHC app will show 200 OK response and the Verizon Networkfleet input count will increase on the Monitor page in GeoEvent Manager.

 

DHCresponse.PNG

NetworlfleetCounter1.PNG

On the DHC app, click the Send button a few more times and notice the Verizon Networkfleet input count increases each time. To test the data flow through GeoEvent Server, create a GeoEvent Service which includes the Verizon Networkfleet input as well as an output such as Write to a JSON File which you can then inspect the content of the file after sending a few more message from the DHC app.

 

Repeat this same test on another computer to verify GeoEvent Server is able to receive the data from other machines. For this, you will need to replace localhost with the actual machine name of the GeoEvent Server instance in the posting URL in the DHC app. If the count values for the Verizon Networkfleet input in the GeoEvent Manager's Monitor page increases after sending more data from the DHC app, then you are successfully configured. If not, first check the firewall rules on the GeoEvent Server instance and ensure communication on port 6180 or 6143 is allowed. If those ports are allowed but you are still not receiving data, there may be other factors related to your network configurations and rules within your organization that may be preventing the data to communicate between machines. In this case, reach out to your organizations network/IT person for assistance.