New Service Designer Capabilities in ArcGIS GeoEvent Server 10.8
Have you ever had to employ a file-based output connector such as Write to CSV File to better understand and troubleshoot what is happening with your real-time data as it's processed in a GeoEvent Service? Or rather, have you ever had to publish a pseudo stream service or feature service just to verify the geometry of your event data is as expected? What happens when you forget to configure an output connector while creating a new GeoEvent Service? What do you do when you realize you need to edit an input in a GeoEvent Service before you're ready to publish it?
If you're like me, these are just a few hurdles you've encountered when working with different elements in the service designer of ArcGIS GeoEvent Manager. It can take time, patience, and a certain level of familiarity to be successful. We've recognized how many steps you must go through to simply and quickly determine what's happening with your real-time data as it's processed in a GeoEvent Service. Additionally, having to navigate to the multiple pages in GeoEvent Manager to create new or edit existing inputs, outputs, site settings, and more can take time. Given these realities, i'm excited to announce many new usability enhancements as well as a new sampling utility available in the service designer at 10.8. These new capabilities will help you be more efficient and effective when defining your real-time services, making the hurdles mentioned above a thing of the past.
So, let's explore some of these new capabilities in more detail!
Working with elements and settings in the service designer
First, we’ve made some exciting functionality and usability enhancements to the service designer in GeoEvent Manager. Many of you are probably familiar with the New Element list where you could previously only add new filters and processors to a GeoEvent Service. With 10.8, you now have the ability to create new inputs and outputs as well as copy existing inputs/outputs and configure them directly in the service designer.
So if you're in a GeoEvent Service, and you forget to create an input or output connector, there's no longer a need to leave the current GeoEvent Service or open another browser tab just to create it. Simply add an input element from the New Element list to the canvas, configure the properties, and save it. It's the same workflow you've become familiar with when adding and configuring both filters and processors.
You may be thinking, “what do I do if I need to edit an input or start or stop that input?” Well, there's now the ability to start, stop, and even edit existing connectors from inside the service designer. Simply right click an input/output you've added to the canvas or one that exists in the element list to access the action menu. For example, in the illustration below, by right-clicking the Receive Flights Data input, you can choose to edit the input's properties, delete the input, fit the input's bounding box to the text, as well as start or stop the input to control the flow of data. By choosing to edit the input properties, a new window will appear allowing you access to all the familiar properties of the input. Should you choose to stop the input, a new status icon on the element will change from green to gray to reflect the stopped state.
Quickly edit, delete, start, and stop an input or output by double clicking the element in the canvas.
While inputs and outputs connectors are certainly important, they're only half the story when it comes to publishing a working GeoEvent Service. There's also several GeoEvent Server site settings that need to be considered and set appropriately, oftentimes, before successfully publishing a GeoEvent Service. That leads to our next exciting enhancement, the ability to access and edit several key GeoEvent Server site settings directly in the service designer. These settings include access to your GeoEvent Definitions, GeoFences, Data Stores, and Spatiotemporal Big Data Stores.
Have you ever realized halfway through configuring a spatial filter that you forgot to configure your GeoFences? Or, maybe you forgot to create the target GeoEvent Definition for a Field Mapper Processor? Rather than leave your GeoEvent Service, and potentially lose the work you put into configuring it, simply access those settings directly in the service designer. From the Site Settings list, you can double click any of the settings available to open and access those particular settings. When you're finished making any necessary updates, just save your changes and continue configuring your GeoEvent Service.
Sample and view real-time data
In addition to the above enhancements to the service designer, i'd like to next introduce you to our newest utility in ArcGIS GeoEvent Server 10.8; the GeoEvent Sampler. As the first of its kind, the GeoEvent Sampler is an embedded utility in the GeoEvent Manager's service designer that allows you to quickly sample, review, and even visualize processed data in real-time as it flows through routes and elements in a GeoEvent Service. No longer is it necessary to spend time and effort configuring different types of ephemeral outputs just to review, visualize, or verify your processed data is as expected.
Unlike GeoEvent Logger or GeoEvent Simulator, which are separate Windows applications, GeoEvent Sampler is embedded in GeoEvent Manager. Therefore, you'll be able to use this new utility in both Windows and Linux environments.
Let’s explore how the GeoEvent Sampler can be used to help you build and/or troubleshoot a GeoEvent Service.
Verifying your schema
Let’s say you want to ensure your Field Mapper Processor is altering the schema of your processed event data correctly. Prior to 10.8, you could write the data emitted from the Field Mapper Processor to an external JSON file. This meant first creating and configuring a separate Write to a JSON File Output Connector. Next, you would need to add that output to your existing GeoEvent Service. Once this was set up, and your outbound data was writing to a JSON file, you would then need to open the JSON file in a text editor to review the data as formatted JSON. While this workflow isn't necessarily difficult to accomplish, it can be time consuming.
Using the new GeoEvent Sampler at 10.8, you can simply select the route that connects the Field Mapper Processor to your next element (e.g. an Output Connector as shown in the illustration below) and sample the event records (formatted in JSON) that are being emitted in real-time on that route.
A sampled event record allows you to confirm the definition and schema are correct.
By never having to leave the service designer, you can quickly confirm whether or not the schema is being updated as expected. If you happen to notice that one of your target definition fields is spelled incorrectly, or that the data type of a field is a string instead of an integer, simply edit the target GeoEvent Definition without leaving the service designer. Remember, GeoEvent Definitions can now be edited directly in the service designer at 10.8 as mentioned above. After making your edits, want to double check that the changes are correct? Just publish the GeoEvent Service and sample the event data on that route again!
Another useful scenario where GeoEvent Sampler could come in handy is if you wanted to compare event data on two routes in a GeoEvent Service. For instance, if you wanted to compare your event data's original schema to the schema after it's emitted from a Field Mapper Processor. First, select and sample the first route that's sending the data into the Field Mapper Processor. This will sample the data right before its schema is transformed by the processor. Next, select and sample the route that's emitting data from the Field Mapper Processor for comparison. This will sample the data after the schema as been transformed. With just a few clicks, you can see the data changing in front of you, in real-time.
Comparison sampling confirms the source and target fields are mapped correctly. For example, the MPH field (left) was renamed to Speed (right).
Verify attribute values
GeoEvent Sampler can also be used to verify the attribute values of your real-time data. Continuing with the flight data example in the illustration above, there is a new field called Speed whose value is in miles per hour (e.g. 500 mph). Let's say you want to change the mph value to kilometers per hour (kph). To do this, you need to use a Field Calculator Processor to multiply the mileage value in the Speed field by 1.609344 to get the kilometers per hour (i.e. Speed * 1.609344). While this is easy enough, how can we quickly verify the conversion is happening before configuring the rest of the GeoEvent Service?
Prior to 10.8, one option would be to create and configure a Push Text to an External TCP Socket Output Connector and view the output in GeoEvent Logger. Data from before and after the Field Calculator Processor could be routed to the output and viewed in GeoEvent Logger. While this is relatively simple to set up, it does take time, just like the previous JSON file example above. Something else worth considering with this type of workflow is data velocity. Depending on the rate at which your real-time data is being received and processed, it could be a challenge to review the data in GeoEvent Logger since it could be constantly updating with new information. You could close the TCP connection in order to allow you to review the data in GeoEvent Logger, but that's just another factor to consider.
Now at 10.8, you can use GeoEvent Sampler and select the route before the Field Calculator Processor to observe the speed data in miles per hour (mph) and then select the route after the processor to view the speed data after its been converted to kilometers per hour (kph). It's important to note that GeoEvent Sampler is not a logging utility, you can only sample a fixed number of GeoEvents on a route (1, 10, or 100 at a time). So, in the case of the flights example, you could quickly sample a single GeoEvent on each route to verify the processor is correctly calculating the speed in kph. There's no need to comb through hundreds of processed GeoEvents to verify the same thing. The graphic below illustrates this.
Comparison sampling of two non-adjacent routes confirm the original MPH field (left) and it's value of 502.0 was correctly converted to kph in the Speed field (right) with a new value of 807.89.
Verify geometry
So far we've covered how GeoEvent Sampler can be used to review and validate your schema and attributes, but what about the geometry of your processed data? Well, included with GeoEvent Sampler is a capability called the Event Viewer. It can be used to display the geometry of processed event data whose GeoEvent Definition geometry field has been tagged with GEOMETRY.
Let's say you're tracking airplanes and using a Buffer Creator Processor to create a buffer for each airplane. You want to ensure the airplanes are being buffered by the correct distance before proceeding with the configuration of the GeoEvent Service. In this case, you need to make sure the point geometry is being changed to polygon geometry. Prior to 10.8, you could verify this a few different ways. First, you could write the buffered GeoEvent to a JSON file and review the geometry object for rings. But even then, how do you know that this geometry will be displayed correctly in a feature or stream service? Unless you're an avid fan of deciphering JSON syntax, you'll likely try to send the data to a feature or stream service to see if it takes. After all, if you can see the geometry of the data display in a web map, you can be confident that buffering is happening, and that the geometry is what you expect. Of course, checking the geometry of a GeoEvent by using a temporary feature or stream service first means creating and configuring those services. Before that can be done, there's GeoEvent Definitions, data store connections, and other factors to consider.
Using GeoEvent Sampler at 10.8, you can now sample the GeoEvents emitted from the Buffer Creator Processor. Once you have a sampled GeoEvent, you can then simply open the Event Viewer to display the geometry. If you choose to sample a single route, the Event Viewer will display the geometry of the sampled records from that route. If you choose to sample two routes for comparison, the Event Viewer will display two map views that show the geometry of the sampled data from both routes respectively. In the illustration below, you can see the point geometry of the source flight data and the polygon geometry of the buffered data output from the Buffer Creator Processor.
The Event Viewer in GeoEvent Sampler, allows you to confirm the valid point geometry of the source event data and the valid polygon geometry of the buffered event data.
These are just a couple ways GeoEvent Sampler can be used to assist with the creation or troubleshooting of your GeoEvent Services. Other example use cases of GeoEvent Sampler not covered here include (but are not limited to) checking datetime values, filtering, regular expressions, construction point geometries, and more.
Hope you enjoy using these new enhancements! If you have ideas for future enhancements, please submit those on the Real-Time GIS place on ArcGIS Ideas.