GeoEvent: Single Stream Service to Rule Them All
NOTE: This blog specifically discusses GeoEvent Stream Services, but could easily be adapted to Feature Service outputs as well.
The Issue
Single Feed for Many Types & Users
I recently ran into a GeoEvent configuration where an input set of events needed to be distributed out to several different layers on a map. The layers were to be distributed among several different groups of users, some with overlapping interests and others were the events should only be seen by one group. An example of this might be a unified county level vehicle feed that includes all county-owned vehicles (public works, ambulance, police, fire, animal control, parcels, etc.) that should be split and shared with users on a selective basis.
NOTE: The diagrams below have omitted anything that isn't specifically relevant (like GeoEvent Services and/or processors).
- The Issue
- Single Feed for Many Types & Users
- Complex Implementation Pattern
- Suggested Implementation
- Push Complexity Downstream
- First: Keep GeoEvent Simple [KGES]
- Second: Leverage Enterprise
- Publish a Stream Service Item
- Third: Create a Stream Service Item View
- Configure the Stream Service Item View
- Share the Stream Service Item View
- Use Stream Service Item View
- Repeat View Creation...
Complex Implementation Pattern
The GeoEvent Server implementation I ran into was exceptionally complex, containing over 20 filters each leading to their own stream service. This solved the issue of splitting the events out to their respective data feed, but did not directly address the situation where some events (e.g. police) should be visible only by certain users. To solve that issue, the customer had chosen to not publish the stream service as an item, but rather to add it to individual web maps directly as a layer from the web (relying on the fact that most users wouldn't be savvy enough to resolve the stream service end point and exploit it).
However, this publishing method proved to be increasing the complexity issue because each of the 20+ stream services was now being used in multiple web maps. This final complexity proved to be the straw that broke the camel's back: Stream service URLs are case-sensitive (at least at 10.7.1 and before). So if a stream service, published as "Stream-Service-n", was added to a web map using a URL containing "Stream-Service-N" it would eventually fail. This failure was hard to debug and identify, but it was finally reproduced (indicated by the red connector in the diagram below).
It should also be noted that this implementation is an expressed anti-pattern. Each filter results in the traffic on the event message bus to double (each connector path has to get a copy of each event). Having lots of filters in parallel like this will cause significant performance degradation for GeoEvent.
Suggested Implementation
Push Complexity Downstream
While the above implementation is probably the first one that comes to mind, I argue that all that complexity should be pushed to the ArcGIS Enterprise. The stream service implementation, based on web sockets, was designed to allow optimized client side filtering, so each client can consume only the events they are interested in. Combine this with Enterprise's implementation of registered items, sharing, groups, and views and you've got a solution that is better suited to handle the complex requirements for who can see and wants to see specific events.
First: Keep GeoEvent Simple [KGES]
As a real-time system GoeEvent's performance and reliability is top priority. As such, it pays to keep a 'keep it simple' mantra in mind when processing your events. In this specific use case, the unified event source should remain exactly that: unified. Assuming the amount of processing required for each event is the same regardless of what group the event belongs to, this is relatively easy.
If some events need to be processed differently, you can either process those events in-line if the outgoing schema will remain the same.
Alternatively, you can create a new GeoEvent Service to subsribe to the unified stream service, applying a filter for the events that must be processed separately, and sending those to the same or a different stream service. Sometimes schemas change, and a second stream service is required (as shown in the diagram below).
Second: Leverage Enterprise
Enterprise is great at managing items, creating views of those items, grouping them for relevance, and sharing them with the right people. We will leverage that to the max here. First though, a note about the Stream Servic URL that you will use to publish it as an item in enterprise.
The URL can be obtained by going to the GeoEvent Server'sREST endpoint, something like:
https://<fqdn>/arcgis/rest/services/
Locate your stream service and click on it to open its REST endpoint, something like:
https://<fqdn>/arcgis/rest/services/<folder>/<stream-service-name>/StreamServer
The URL in the address bar is the URL you need below.
IMPORTANT NOTE: The <stream-service-name> in this URL is case-sensitive. Your name, including case, must match exactly the name of the stream service created by GeoEvent.
Publish a Stream Service Item
Assuming your stream service isn't published automatically to your enterprise, you can add it using the 'Add Item > From the web' button.
Press the 'Add Item' button |  |
On the new items page, navigate to the 'Visualization' tab at the top. Modify the default symbology if needed, for example if all of your events will be symbolized by arrows pointing in the direction of the heading. Press the 'Save Layer' button to save your default symbology changes.
Third: Create a Stream Service Item View
Now press the 'Save As New Layer' button at the top right; this will create a targetd event stream service layer layer:
- Title: Give it a name that represents the type of data it is meant to represent.
- Tags: Same thing for tags
- Summary: Provide a more detailed summary of the data you are presenting in this stream service item view
- Save in Folder: Choose a folder to save the data in
Press the 'Save' button. Note that this does not create a copy of the stream service. Instead, it creates a view on top of the stream service. This is simply a different subscription to the same stream service. This new, separate subscription will be able to maintain its own unique settings for consuming and displaying the stream service data.
Configure the Stream Service Item View
Once you've saved the new view, navigate to the Visualization page at the top right. Select the filter button and filter the view to only the data it is meant to represent. The settings in this filter are a replacement for the filter settings in the GeoEvent Server. Also be sure to configure the symbolization, pop-up, and clustering options. Once you have your specialty view layer configured, press the 'Save Layer' button.
Share the Stream Service Item View
Once you have your view layer configured, it is time to share it with your users. To do that, navigate back to the Overview tab on the top right and press the 'Share' button. You can share this view with any groups within your enterprise, the entire enterprise, or the public. Notice that we didn't share the original stream service item; that was on purpose to show that the underlying stream service can be private, while specific views of that service can be shared.
Use Stream Service Item View
Once the item is shared with a group, you can add it to as many web maps as needed within that group. Users who belong to that group will also be able to search, discover, and use the view layer in their own personal maps.
Repeat View Creation...
Repeat Step Three above as many times as needed to complete your system. This may feel like you are trading complexity at the back end of the system for complexity at front end, but the enterprise was built to manage and control the complexity of creating views and sharing them.
