Latest Contributions by EricIronside

Most of the time in GeoEvent we are trying to get the events out the door as fast as possible.  But in a few rare cases, we need to pause for a moment. This blog details a couple of event processing options that delay events for a specific benefit.    Note: The custom processors mentioned here were created by Esri staff, but are considered custom components and not supported officially by the GeoEvent product team.  If you like these processors, and want to see them included either in the GeoEvent gallery or directly in the GeoEvent product, please create an Idea to vote on and leave a comment about it on this article.  If you find an bug or want to request an enhancement to either of these custom processors, please create an issue on the respective GitHub page.     Delay Processor As I said above, most of the time we want events out the door as fast as possible. But sometimes, it helps to delay things a bit. For example, a public facing map of street sweepers or snow plows will show anyone that is interested exactly where your vehicles are. But for various reasons, it may not be a good idea to display their exact location. Instead, you might want to show the public where those vehicles were 15 minutes ago. This strategy balances safety of your drivers and assets with the need for letting the public know the current status of a situation.    For this scenario, I would use the Delay Processor published here:        https://github.com/mshareghi/geoevent-delay-processor/releases/tag/v1.0    This processor takes a delay time (in practically any time unit you might need) and adds it to the received time of each event.  The events are then put into a delay queue that will release them from the processor once they've waited out their delay.  In our example above, if you introduce the Delay Processor just before writing your data out to that public Feature Layer on ArcGIS Online, you can set it to delay each event by 15 minutes.    Temporal Ordering Processor In this scenario, you might be polling a data provider that returns location events out of temporal order. The most common example of this is:   Delayed/Batch reporting - A vehicle may go offline for many hours (maybe it is visiting remote facilities outside the range of cell coverage). During this offline time, the device in the vehicle may cache location data until it re-connects to the cellular network. The slug of location events provided by the AVL API may be in any order.   In this situation, it is impossible to do any reasonable stateful analysis on the data (e.g. "What time did the vehicle enter/exit a facility?").  Wouldn't it be great if we could tell GeoEvent to temporally order these events and then release them in the correct order and time space?    An updated Delay processor created by me can do just that:       https://github.com/eironside/geoevent-delay-processor/releases/tag/1.4   In this case, the delay must be set to the longest period of time your data might be out of order by.  In the example case above, the vehicles might have been offline for as much as 8 hours. So the delay time would be set to 8 hours (or maybe 9 just to be sure we've got everything before proceeding with processing).  However in other cases, your data may only be out of order by a few minutes, so in those cases you can definitely reduce the delay down to something like 3 or 4 minutes.  Hopefully, an accurate analysis of data is more important than the introduced delay.   Note: For this scenario, the other properties included with this processor should be set as follows: Allow Duplicates? Yes, Include TRACK_ID? No   Unique Temporal Ordering Processor Finally, you may have a data provider that is giving a batch of data that covers a specified time span (e.g. last 60 seconds of data). In most of these cases, there's no guarantee the data will be in the correct temporal order or even unique (you may end up getting the same event over and over again).  An example of this might be:   A data provider API only allows you to request data for the last 5 minutes - You can request the data more frequently, but it will always return the last 5 minutes of location information for each vehicle.  If each vehicle is reporting every 30 seconds, there may be as many as 10 events per vehicle in each response (or there may be none if the vehicle is off).  The returned data from each request may or may not be temporally ordered correctly.   In this example, we see a similar problem to the Temporal Ordering Processor above with an added twist. Each batch of data could have as many as 9 repeat events.  In this case, it would be good to be able to identify the repeat events and discard them since they are already in our delay queue.     Use the updated Delay processor created by me but tell it to discard duplicate events:         https://github.com/eironside/geoevent-delay-processor/releases/tag/1.4   For this example, you would set the delay to be something slightly larger than your polling rate (e.g. if you are polling every 90 seconds, set the delay to 120 seconds).  You would then set Delay Time Field to TIME_START, Allow Duplicates? to No and Use TRACK_ID? to Yes.  Each event record's unique key would then become "<timestamp>_<trackid>". Any duplicate events with a key that matches an event already processed would be dropped (namely the same TRACK_ID and TIME_START). ... View more

Consuming New Data in GeoEvent This blog consolidates some notes from other blogs about creating new GeoEvent Definitions when receiving new data. the sections are a bit redundant, but I'll leave it that way since it is important information.   JSON/XML Object Name Usually you don't know what kind of data to expect and just need to receive some data before deciding how to parse it.  In that case I usually leave the Object Name empty and let GeoEvent try to parse the entire message.  Once you know what the data schema looks like, you can come back and modify this property to target the specific section of the incoming data you want to turn into events.   Also note that you may end up identifying one or more sections of the incoming data you want to parse into separate/different events. In that case I would create a second GeoEvent Input for the second section of data I want to parse. Auto-Created GeoEvent Definitions I have a habit of making copies of any GeoEvent Definitions that are auto-created for me by GeoEvent modifying the item that auto-created the definition to point to my copy. This allows me to know the exact schema of the data at all times and avoid missing or changing definitions. Because of this, any time I use an item that is automatically going to create the GeoEvent Definition for me, I add a postfix "-auto" to the end of the definition name.  This allows me to do two things: Identify any GeoEvent Definition that was automatically created (and hence may disappear or change) Copy it and rename it without the -auto on the end, so I own that schema and it will be permanent.   Lock Down Your GeoEvent Definitions Once you receive some data and GeoEvent has helped you determine the schema, you can lock down that schema so it doesn't change and the GeoEvent definition is not transient.   When you have the Create GeoEvent Definition set to Yes, the definition it creates is transient and owned by the input. This means the event can change or even disappear if the input is stopped. Those are things we don't want to happen without us explicitly doing it. To get around this: Have the input create the definition for you, but name it something like "MyWebhookName-in-auto". After the definition is created by the input, make a copy of it.    Name the copy something like "MyWebhookName-in" (take the "-auto" off) Back in your input, change Create GeoEvent Definition to No and select the new definition you just created.   Learning Mode Learning mode can be very useful, but dangerous if left on.  In some cases where the data you are listening to could be delivered in either partial or different schemas, it might be useful to leave learning mode on for a bit to capture the schema of all the data you might receive. For example, If you are receiving events that omit a field when that field is null. If you are receiving a mix of different events on a single input feed.   Once you are confident that you've received enough events to complete the schema(s) you are listening for, turn learning mode off. Not turning learning mode off has bit me hard several times so DON'T FORGET TO TURN OFF LEARNING MODE (yelling at myself).    After you have used learning mode to discover your data schema, make one or more copies of the learned schema (see Auto-Created GeoEvent Definitions above). For each copy, delete any unused/needed fields (but make sure you don't change the name!) and save the copy with a descriptive name (like appending "_IN" to the end). Once you've made your copy, make sure you go back to your input and change Create GeoEvent Definition to No and select your copy of the definition. ... View more

One of the most powerful things you can do with GeoEvent is monitor spatial events to determine how your real-time assets are interacting with each other.  But sometimes, using the spatial operators in the various processors provided by GeoEvent can be intuitively challenging. Below are some patterns (and anti-patterns) that might be useful. Cached vs. Non-Cached  Spatial Operator Most of the spatial operators are non-cached, meaning that they don't require any knowledge of where an asset has been. Examples of non-cached operations include inside/outside, intersects, and disjoint.  Evaluating these conditions can be done immediately and without any other information other than two geometries. On the other hand, cached operators require more information than just one event can provide.  An example of a cached operator is Enters. In order to determine if an event has entered a goefence, the operator needs to know where the event was before the current event. In order to perform the operation, the operator must cache the previous location of each TRACK_ID so that it can know where that asset has been. Filters Filters are special processors that allow events to pass or not based on a predetermined set of conditions. Those conditions can be based on either attribute values (speed > 20) or on spatial conditions (inside a geofence).  Filters can also evaluate the cached spatial conditions such as Enter/Exit.   The general guideline is to filter events as soon as possible in your GeoEvent Services to reduce the overall processing load. Combine this with the fact that evaluating spatial conditions can impose a slightly larger processing load when compared to determining attribute conditions ("speed > 65" will probably be computationally easier than "point inside polygon") and it makes sense to try to remove events as early and easily as possible. An example of a good practice for using a filter to limit your spatial analysis might be: Filter-GeoTagger Anti-Pattern But one must keep in mind that using a cached spatial operator within a filter will only pass the event that has met the conditions.  This may impact processors down-stream of the filter, if they are also implementing a cached spatial operation. This brings us to our first Anti-Pattern: Filters remove events that some cached spatial operators need. Let's say you want to process truck locations that are entering or exiting a yard.  Based on your knowledge, you know that there will be a lot of GPS locations that have nothing to do with entering or exiting the yard so you place a filter at the beginning of your service. But you also want to know what yard the truck entered/exited, so you use a GeoTagger to capture the name.  The resulting service might look something like this: You will find that this does not work for the following reason: The spatial operation in the GeoTagger is a cached operation, so it requires two (2) events to determine the enter/exit condition of a truck (one outside and one inside). However, the filter at the beginning is only allowing one of the two required events to pass. Combine this anti-pattern with the global settings "First Event Triggers Enter = TRUE" and "First Event Triggers Exit = FALSE", and you will find it causes unexpected behavior and should be avoided. Anti-Pattern Details The filter is masking event records the GeoTagger needs in order to correctly determine Enter / Exit. Each node maintains its own cache of observed events (one reason stateful operations like “enter” and “exit” are problematic). The filter “ENTER” determines an event is outside the AOI (so ENTER == FALSE) and the event is discarded. We don’t care that the filter discarded the event in this case because nothing interesting happened (we’re not entering an AOI). But when the next event intersects the AOI and its previous track point was just outside, the filter “ENTER” determines ENTER == TRUE and passes just the one event along. What we have to understand is that the “Tag on Enter” GeoTagger, at this point, has never seen an event record with this track identifier. The “First Event Triggers Enter” property (a Global Settings property) allows the GeoTagger to evaluate ENTER == TRUE. So we get the tagged event we expect. The same exact evaluation is performed when the event record enters the overlapping area for the first time: The weirdness starts when the track point is first observed exiting the geofence “Hippo”. The GeoTagger output has a null value in the geofence’s name, even though the Filter determined that an exit has occurred. Why? Because the “Tag on Exit” GeoTagger never saw an event record inside “Hippo” and the global setting “First Event Triggers Exit” default to False. The EXIT filter saw the previous event, inside, and was therefore able to determine EXIT = TRUE … but all the previous events which were not “exiting” were discarded by the filter so the GeoTagger never saw them. The GeoTagger therefore evaluates EXIT == FALSE and places a null value into the event record’s field as the name of any geofence the event has found to have exited. When the event moves outside “Campus” the “Tag on Exit” GeoTagger has seen an event record inside “Campus” … when it was just exiting “Hippo” … so it is happy enough to evaluate EXIT == TRUE and enrich the event with the name of the geofence the event record was last observed inside and is now outside. The real weirdness starts when you run the same eight track points through the GeoEvent Service. This time the “Enter” branch at the top seems able to determine that the event is entering something but cannot name the geofence being entered. This is again explained if we recognize that we have not done anything that would clear either the ENTER Filter’s cache or the “Tag on Enter” GeoTagger’s cache (like republishing the GeoEvent Service). The “Tag on Enter” GeoTagger has exactly two observations; event record locations inside “Hippo” and inside both “Hippo” and “Campus” when the event records entered those areas for the first time during the last iteration. All of the other received event records were discarded when the ENTER filter determined “not entering”. So as far as the “Tag on Enter” GeoTagger is concerned, the event is still inside the area “Hippo” when the second report for the second iteration is received. The GeoTagger determines ENTER == FALSE because it has seen records with this track identifier before, so the “First Event Triggers Enter” property does not apply. Focusing, then, on the last two outputs for the second iteration … analytics for exiting are on a completely separate branch of the GeoEvent Service. That last time the “Tag on Exit” GeoTagger saw a track point from this stream was when it had just exited both geofences and was disjoint all geofences. So when it gets the next “exiting” event (the 6th in the series of eight) it determine EXIT == FALSE because it has not observed the event inside at any point between its last observation and this observation. Therefore the second-to-last message output (again) has a null for the geofence name. But when the event finally exits the two geofences for the last time the “Tag on Exit” GeoTagger has seen the event inside “Campus” so it’s at least able to tag that geofence’s name into the exiting event record. GeoTagger-Filter Pattern To fix this, you need to reverse the order of the GeoTagger and filter.  The GeoTagger will capture the name of the yard that was entered/exited (and a null value if the truck is not entering/exiting a yard). The filter will remove any events that have null enter/exit information. Computationally, this pattern has the same number of initial spatial operations (to perform the enters/exits operation), but substitutes the second spatial operation for a simple null attribute check.  Cached Spatial Operations + NULL As you might expect, the cached spatial operators are reliant on continuous good data.  Unfortunately some data feeds are not always accomodating.  For example, if you have a data feed that occaisionally provides a NULL geometry, then this NULL will be cached.  What this actually means is that the last known location for the event's TRACK_ID will be removed and nothing will take its place. So the spatial operation will think it has never seen that TRACK_ID before.  This will cause the spatial operator to rely on the global settings to determine what to do with the next valid location for that TRACK_ID. Additional Discussions For some additional discussions of the Enter/Exit operations, please see RJ Sunderman‌'s responses to the following conversations: How to get one single entry message and one single exit message once a ship enters in a geofence area  Geotagger processor gives all geofences with exit option  Cached Operation Settings Because GeoEvent must maintain a cache of events to determine the condition of some spatial operators, they are subject to the global settings related to caching. A good discussion of this can be found at the following links:  GeoEvent Documentation Managing global settings—Administer(10.7.1) | ArcGIS Enterprise (see "Geofence Manager Settings") The following blog https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2016/10/26/using-a-geotagger-processor-with-more-than-1000-unique-track-identifiers   by Chris Beyett‌ ... View more

You can use the Admin API of GeoEvent to create or update new GeoEvent Definitions, but be sure to use the right JSON format.  When the API encounters an error when importing your definition, the error message can be a bit misleading/confusing.  To clarify this a bit, I'm collecting examples of what might go wrong on this page. I will update it if I find more. Cannot deserialize instance ... When posting your JSON to the API, you may encounter the following error message: Cannot deserialize instance of `com.esri.ges.jaxb.geoeventdefinition.GeoEventDefinitionWrapper` out of START_ARRAY token\n at [Source: (org.apache.cxf.transport.http.AbstractHTTPDestination$1); line: 1, column: 1] In this case it is tempting to see the word token and think security issues.  However, the key to this error message is the first part Cannot deserialize....  The JSON parser has tried to parse your JSON into a GeoEvent Defintion and failed.  The token it is referring to is a parsing/deserializing term related to token/value pairs.  In my specific case, my JSON was valid (as confirmed by an independent JSON LINT) so what could possibly be the problem?? It turns out my JSON GeoEvent Definition was encapsulated in square brackets, indicating I was posting an array of GeoEvent Definitions.  [ { "guid": "12345678-9abcdefgh-1234-abcdefghijkq", "name": "c-in", "owner": "admin", "accessType": "editable", "fieldDefinitions": [ { "name": "cId", "type": "String", "cardinality": "One", "fieldDefinitions": [] }, { "name": "geometry", "type": "Geometry", "cardinality": "One", "fieldDefinitions": [], "fieldDefinitionTag": [ "GEOMETRY" ] } ] } ]‍ The parser attempted to put the array object into a single GeoEvent Definition, causing the failure (it found a JSON Array token rather than a JSON Object token).  The quick fix was to remove the outside square brackets and everything worked as expected. ... View more

Webhooks for ArcGIS for Enterprise were released a while ago so this post will describe how to subscribe to a Webhook from Enterprise (Portal for ArcGIS).  I will leave it up to you to decide what to do with them...  Please note that the payload data contained within a webhook can be variable. You may need to complete this exercise several times, once for each type of webhook payload you wish to consume.  Webhooks Webhooks allow you to register your "payload processor" application [AppA] with another application [AppB].  During the registration process, you tell AppB to send webhook payloads to AppA at a specific URL.  When AppB has some sort of trigger event that it needs to send to AppA, it creates a payload with information about that event (typically in JSON format) and sends that payload to AppA via HTTP to the URL registered above (typically a POST).    In ArcGIS for Enterprise there are a number of webhooks you can register for including item creation/deletion, sharing events, and group events.  More information on creating a webhook and what kind of trigger events you can register a webhook for can be found in the Enterprise documentation. GeoEvent Webhook Input As I mentioned above, a webhook URL is simply an address that an application listens on to receive payload data.  Since that data is most likely JSON, we have a GeoEvent input ready to go OOTB. NOTE: In GeoEvent Manager, set the Log Level to TRACE for the logger com.esri.ges.adapter.genericJson.JsonInboundAdapter while you go through this post so you can see the GeoEvent logs indicating things are working.  Don’t forget to reset the Log Level back to WARN after you are done testing. In GeoEvent Manager, navigate to Services > Inputs and press the Add Input button. Search for "Receive JSON” and select Receive JSON on a REST Endpoint. On the Creating Input – Receive JSON on a REST Endpoint page, modify the properties as needed and press the Save button. Name: [Unique name of this webhook listener] JSON Object Name: [The payload json object name you are looking to monitor; leave empty if you don't know]* Create GeoEvent Definition: Yes** GeoEvent Definition Name (New): [MyWebhookName-auto]*** Advanced:  Acceptable MIME Types (Server Mode): application/json Learning Mode: No**** After you save the input, re-open it. It now will display a URL parameter. This URL parameter is the “Payload URL” that the Webhook will need for it to POST data to. Copy that URL property value and save it. Create the Webhook In a browser, go to your Portal for ArcGIS sharing URL and log in as an administrator: https://<yourServer>/portal/sharing/rest/login Navigate back to the home directory https://<yourServer>/portal/sharing/rest and Navigate to Portals > Self > Webhooks. The webhooks link appears at the very bottom of a very long page. If you don’t see webhooks at the bottom, then you are not logged in or not logged in as an administrator. Press the Create Webhook link, modify the properties, and press the Create Webhook button Name: [Give it a name similar to the input name in GeoEvent] Payload URL: Paste the input URL you copied from GeoEvent above. A list of Supported trigger events can be found here  Back on your GeoEvent Manager > Logs page you should notice some log messages indicating the webhook properly registered: On your input, if you told GeoEvent to create the GeoEvent Definition for you and you set Learning Mode to No then you will need to delete the event it created because the schema for the webhook registration is not going to be correct. Navigate to GeoEvent > Site > GeoEvent > GeoEvent Definitions and delete the definition your webhook input created. Testing the Webhook For the test run below, it assumes you selected ‘Send me everything’ or configured the events to be /items or /items/add. If you selected another event, use that to test your webhook. In a Browser, log into your Portal Home and navigate to your Portal’s Content directory. Press the Create button and select Feature Layer. Choose a feature layer template from the list (it doesn't matter which one) and press the Create button. Press the Next button until you get to the properties page. Enter a Title, Tags, and a Summary then press the Done button. After you press the Done button, go back to GeoEvent Manager > Logs and verify the Create Item webhook was received. NOTE: Don't forget to reset the Log Level to WARN for the logger com.esri.ges.adapter.genericJson.JsonInboundAdapter after you are done testing. NOTES: Consuming New Data in GeoEvent This section deserves its own blog, but I will put a little detail here until I can get to a full blog on it. * JSON Object Name Usually you don't know what kind of data to expect and just need to receive some data before deciding how to parse it.  In that case I usually leave the JSON Object Name empty and let GeoEvent try to parse the entire JSON message.  Once you know what the data schema looks like, you can come back and modify this property to target the specific section of JSON you want to turn into events. Also note that you may end up identifying one or more sections of the incoming JSON you want to parse into separate/different events. In that case I would create a second GeoEvent Input and register a second webhook to have data sent to it. ** Create GeoEvent Definition Once you receive some data and GeoEvent has helped you determine the schema, you can lock down that schema so it doesn't change and the GeoEvent definition is not transient. When you have the Create GeoEvent Definition set to Yes, the definition it creates is transient and owned by the input. This means the event can change or even disappear if the input is stopped. Those are things we don't want to happen without us. To get around this: After the definition is created by the input, make a copy of it.    Name the copy something like "MyWebhookName-in" (take the auto off) Back in your input, change Create GeoEvent Definition to No and select the new definition you just created. ** Auto-Created GeoEvent Definitions I have a habit of making copies of any GeoEvent Definitions that are auto-created for me by GeoEvent modifying the item that auto-created the definition to point to my copy. This allows me to know the exact schema of the data at all times and avoid missing or changing definitions. Because of this, any time I use an item that is automatically going to create the GeoEvent Definition for me, I add a postfix "-auto" to the end of the definition name.  This allows me to do two things: Identify any GeoEvent Definition that was automatically created (and hence may disappear or change) Copy it and rename it without the -auto on the end, so I own that schema and it will be permanent. **** Learning Mode Learning mode can be very useful, but dangerous if left on.  In this case where the webhook you are listening to could be capturing different schemas, it might be useful to leave learning mode on for a bit to capture the schema of all the data you might receive via the webhook payloads. For example, if you had turned learning mode on, you would not have had to delete the first GeoEvent Definition created when the webhook sent the first "webhook registered" payload.   Once you are confident that you've received a webhook for each event you are listening for, turn learning mode off. Not turning learning mode off has bit me hard several times so DON'T FORGET TO TURN OFF LEARNING MODE (yelling at myself).  After you have used learning mode to discover your data schema, make sure you go back and change Create GeoEvent Definition to No (as discussed above). ... View more

This article discusses a targeted way to debug erroneous records created by GeoEvent in a hosted feature layer.  For more general details on configuring and using GeoEvent logging system, please see RJ Sunderman‌'s series of blogs on the topic: https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2019/06/15/debug-techniques-configuring-the-application-logger?sr=search&searchId=57faaa8b-79a7-4909-b930-26e6a97c7fa9&searchIndex=1 The Situation We have a GeoEvent Service that is reading in events from a hosted feature layer, attempting to enrich the data, filtering any data that didn't get enriched, and writing them out to another hosted feature layer: The filter is attempting to pass only events that have a valid session_id Obviously, it is expected that the data table in the target hosted feature layer will not contain any features with a NULL session_id.  Yet, when checked there are occaisionaly records that get through the filter.   Debugging Follow the steps below to debug this issue: In the configuration file C:\Program Files\ArcGIS\Server\GeoEvent\etc\org.ops4j.pax.logging.cfg Set the log4j2.appender.rolling.strategy.max to something large like 200 In GeoEvent Manager Create a new GeoEvent Input Poll an ArcGIS Server for Features that reads records from your target Hosted Feature Layer (the feature layer that is getting records with null that you don't want to see) Be sure to set the where clause to monitor for the error condition like session_id is null Create a new GeoEvent Output Send an Email that will send a notice to you that an erroneous record has been stored in the Hosted Feature Layer Add your email to the Sender and Recipient list Set the message format to HTML In the body, you should consider adding the important fields from the failure event like SessionID: ${session_id} Include a link to the REST end point query method so you can quickly evaluate the invalid record(s) like the following:   <a href=”https://<YourServerName>/arcgis/rest/services/Hosted/Last_Known_Locations_HFS/FeatureServer/0/query?where=session_id+is+not+null&outFields=*”>Click Here To See Errors</a> Create a new GeoEvent Service called “Monitor for Errors” Add your Feature Service Poll input and Email output from above then connect them up Press the Publish button Create a new GeoEvent Output Write to a JSON File to write data to disk Name it file-json-out-before Set the Filename Prefix: before Modify other settings as needed Create a new GeoEvent Output Write to a JSON File to write data to disk Name it file-json-out-after Set the Filename Prefix: after Modify other settings as needed In your existing GeoEvent Service that is currently processing data Place the file-json-out-before before your final filter Place the file-json-out-after after your final filter Press the Publish button   Set the LOG LEVEL on the following loggers to TRACE com.esri.ges.transport.featureService.FeatureServiceOutboundTransport com.esri.ges.httpclient.http Wait for an email   While You Wait... While you are waiting for an issue to occur, keep an eye on your disk space/usage for the files being written out: Periodically delete old karaf#.log files in your C:\Program Files\ArcGIS\Server\GeoEvent\data\log directory. Periodically delete old before#.json and after#.json files in the output directory you specified. You've Got Mail! Once you get an email indicating an error has occured: Stop all GeoEvent Inputs, Outputs, and Services if you can to make sorting out the logs easier (or stop the GeoEvent Windows Service while you collect the files). Collect the files from the following locations. You are looking for files that contain data before and after the timestamp on the email. C:\Program Files\ArcGIS\Server\GeoEvent\data\log\karaf#.log <Your JSON Output File Directory>\before#.json <Your JSON Output File Directory>\after#.json You will want to collect logs from your ArcGIS Server during this time as well. If the failure record contains any sort of identifiable information you can use that to locate the event in the files above. If not, you may have to resort to sorting through the timestamps until you find the exact time the event went through the system. See RJ's blogs mentioned at the top of this page for tips/tricks on analyzing the log files. See Also: Debug Techniques - Configuring the application logger blog by RJ Sunderman ... View more

As you may know, the GeoEvent Manager is reliant on the Enterprise to create the Map and/or Feature Service when connecting to Spatio-Temporal Big Data Store [STBDS or just BDS] data sources.  In most cases, the creation of these services goes smoothly and everything just works. But on occaision, the synchronization between the two systems doesn't go as it should because of timing issues.  This entry walks through one of those situations and a simple fix. Architecture I have an environment consisting of an ArcGIS Enterprise [Portal for ArcGIS, ArcGIS Server], Data Store [Tile Cache, Relational, and STBDS], and GeoEvent Server.  The GeoEvent Server is federated with my enterprise, but I don't think that really makes a difference (I've seen this issue on system with and without federation of the GeoEvent Server). Create Data Source In GeoEvent Manager, I navigated to Services > Outputs and pressed the Add Output button. I selected the Add a Feature to a Spatiotemporal Big Data Store output and created the output using most of the defaults.  While on this dialog, I press the Create Data Source button.  On the resulting Create Data Source dialog, I accepted most of the defaults and pressed the Publish button. Once the user interface returned, I added the output  to a GeoEvent Service, connected it up and data appeared to be flowing to my output. Missing A Service? In order to review/change my Data Source's properties I navigated Site > GeoEvent >Spatiotemporal Big Data Stores. I was presented with the following:   At first glance I feared it had failed to create the Map Service. But I wanted to be sure it had not been created before I created a new one. So I consulted the Portal Content and was surprised to see both of my services listed there: I also looked into the ArcGIS Server REST services and found them to be there and functioning properly: Finally, I added them to a web map. By now, data had added to the underlying feature class and the map displayed properly. Missing Service Fix Seing my services on the Enterprise made me suspect that communication between GeoEvent and the Enterprise had somehow gone wrong.  Returning to GeoEvent Manager, I navigated to Site > GeoEvent > Data Stores, located my connection to the enterprise (in my case it was default) and pressed the Synchronize button. Doing the synchronization caused GeoEvent to update its cache of content on the Enterprise, including the Map and Feature Services related to my Data Source. Now when I go to Site > GeoEvent >Spatiotemporal Big Data Stores my Data Source lists the appropriate services: The Issue When GeoEvent sends the request to create the Map and Feature Services, it will only wait so long before timing out.  Because of this, if the response from Enterprise is delayed for any reason, GeoEvent will continue on without the service information.  Synchronizing the Data Store in GeoEvent Manager causes GeoEvent to re-cache the content on the Enterprise, including the services related to the BDS Data Source. ... View more

NOTE: This blog specifically discusses GeoEvent Stream Services, but could easily be adapted to Feature Service outputs as well. The Issue Single Feed for Many Types & Users I recently ran into a GeoEvent configuration where an input set of events needed to be distributed out to several different layers on a map. The layers were to be distributed among several different groups of users, some with overlapping interests and others were the events should only be seen by one group. An example of this might be a unified county level vehicle feed that includes all county-owned vehicles (public works, ambulance, police, fire, animal control, parcels, etc.) that should be split and shared with users on a selective basis.  NOTE: The diagrams below have omitted anything that isn't specifically relevant (like GeoEvent Services and/or processors). Complex Implementation Pattern The GeoEvent Server implementation I ran into was exceptionally complex, containing over 20 filters each leading to their own stream service. This solved the issue of splitting the events out to their respective data feed, but did not directly address the situation where some events (e.g. police) should be visible only by certain users.  To solve that issue, the customer had chosen to not publish the stream service as an item, but rather to add it to individual web maps directly as a layer from the web (relying on the fact that most users wouldn't be savvy enough to resolve the stream service end point and exploit it). However, this publishing method proved to be increasing the complexity issue because each of the 20+ stream services was now being used in multiple web maps.  This final complexity proved to be the straw that broke the camel's back: Stream service URLs are case-sensitive (at least at 10.7.1 and before).  So if a stream service, published as "Stream-Service-n", was added to a web map using a URL containing "Stream-Service-N" it would eventually fail. This failure was hard to debug and identify, but it was finally reproduced (indicated by the red connector in the diagram below).   It should also be noted that this implementation is an expressed anti-pattern.  Each filter results in the traffic on the event message bus to double (each connector path has to get a copy of each event). Having lots of filters in parallel like this will cause significant performance degradation for GeoEvent.  Suggested Implementation Push Complexity Downstream While the above implementation is probably the first one that comes to mind, I argue that all that complexity should be pushed to the ArcGIS Enterprise. The stream service implementation, based on web sockets, was designed to allow optimized client side filtering, so each client can consume only the events they are interested in.  Combine this with Enterprise's implementation of registered items, sharing, groups, and views and you've got a solution that is better suited to handle the complex requirements for who can see and wants to see specific events. First: Keep GeoEvent Simple [KGES] As a real-time system GoeEvent's performance and reliability is top priority. As such, it pays to keep a 'keep it simple' mantra in mind when processing your events.  In this specific use case, the unified event source should remain exactly that: unified. Assuming the amount of processing required for each event is the same regardless of what group the event belongs to, this is relatively easy. If some events need to be processed differently, you can either process those events in-line if the outgoing schema will remain the same. Alternatively, you can create a new GeoEvent Service to subsribe to the unified stream service, applying a filter for the events that must be processed separately, and sending those to the same or a different stream service.  Sometimes schemas change, and a second stream service is required (as shown in the diagram below). Second: Leverage Enterprise Enterprise is great at managing items, creating views of those items, grouping them for relevance, and sharing them with the right people. We will leverage that to the max here. First though, a note about the Stream Servic URL that you will use to publish it as an item in enterprise.  The URL can be obtained by going to the GeoEvent Server'sREST endpoint, something like:            https://<fqdn>/arcgis/rest/services/  Locate your stream service and click on it to open its REST endpoint, something like:       https://<fqdn>/arcgis/rest/services/<folder>/<stream-service-name>/StreamServer  The URL in the address bar is the URL you need below. IMPORTANT NOTE: The <stream-service-name> in this URL is case-sensitive. Your name, including case, must match exactly the name of the stream service created by GeoEvent. Publish a Stream Service Item Assuming your stream service isn't published automatically to your enterprise, you can add it using the 'Add Item > From the web' button. Type: ArcGIS Server Web Service URL: The stream service REST endpoint URL from above Title: modify the title as needed Tags: add tags as needed Press the 'Add Item' button On the new items page, navigate to the 'Visualization' tab at the top.  Modify the default symbology if needed, for example if all of your events will be symbolized by arrows pointing in the direction of the heading.  Press the 'Save Layer' button to save your default symbology changes.  Third: Create a Stream Service Item View Now press the 'Save As New Layer' button at the top right; this will create a targetd event stream service layer layer: Title: Give it a name that represents the type of data it is meant to represent. Tags: Same thing for tags Summary: Provide a more detailed summary of the data you are presenting in this stream service item view Save in Folder: Choose a folder to save the data in Press the 'Save' button. Note that this does not create a copy of the stream service. Instead, it creates a view on top of the stream service. This is simply a different subscription to the same stream service.  This new, separate subscription will be able to maintain its own unique settings for consuming and displaying the stream service data. Configure the Stream Service Item View Once you've saved the new view, navigate to the Visualization page at the top right. Select the filter button and filter the view to only the data it is meant to represent. The settings in this filter are a replacement for the filter settings in the GeoEvent Server.  Also be sure to configure the symbolization, pop-up, and clustering options.  Once you have your specialty view layer configured, press the 'Save Layer' button. Share the Stream Service Item View Once you have your view layer configured, it is time to share it with your users. To do that, navigate back to the Overview tab on the top right and press the 'Share' button.  You can share this view with any groups within your enterprise, the entire enterprise, or the public. Notice that we didn't share the original stream service item; that was on purpose to show that the underlying stream service can be private, while specific views of that service can be shared. Use Stream Service Item View Once the item is shared with a group, you can add it to as many web maps as needed within that group.  Users who belong to that group will also be able to search, discover, and use the view layer in their own personal maps.  Repeat View Creation... Repeat Step Three above as many times as needed to complete your system. This may feel like you are trading complexity at the back end of the system for complexity at front end, but the enterprise was built to manage and control the complexity of creating views and sharing them. ... View more

Under very specific situations, the import or synchronization of GeoFences within GeoEvent will fail. This blog details the symptoms and provides a workaround. The Issue: Failing to Import/Synchronize GeoFences I recently attempted to import some GeoFences from a Feature Service using a field other than the ObjectID field as the Name Field. In my case the Category Field was hardcoded to "HEX" and the Name Field was a string field called 'Name'. the error I get back is a java.lang.NullPointerException. Looking in the logs, I notice the following error: An unexpected error has occurred while updating the GeoFences from server "Default" in service "WPHex". java.io.IOException: Error accessing data store at https://server.blah.com:6443/arcgis/. : 400 : Unable to complete operation. This error message is telling us that GeoEvent failed to get the GoeFence features from the web service you selected.  If you select the ObjectID for either the Category or Name Field the request will work.  However, if you do not select ObjectID for either of these fields the web service may fail. The Cause: Setting Max Record Count on Feature/Map Service The reason the import above is failing is an issue in the web service (BUG-000126618): When we set the MaxRecordCount property to something other than 1000 any request for more than 1000 features will fail when the ObjectID is not included in the request.  That is a very long sentance, so lets break that down: - A feature or map service is published with more than 1000 features in it. - The web service is published with a MaxRecordCount > 1000 (in this case 2000) - The GeoEvent request for features includes the Name field, but not the ObjectID field (Category Field = "HEX" and Name Field = Name) - The web service fails to return any features, thus GeoEvent has no GeoFences to import. Workarounds Workaround 1: Set Max Record Count = 1000 The first workaround is to set the MaxRecordCount property on the Map/Feature Service to 1000. For whatever reason, the web service works perfectly fine so long as this property is set to 1000. Once you set it to anything larger than 1000, the service fails to return features. Workaround 2: Use ObjectID as Name Field The second workaround is to use the ObjectID as the name field in GeoEvent GeoFence Import/Synchronization.  Using ObjectID, it doesn't matter what the setting is on the service for MaxRecordCount. ... View more

DISCLAIMER: The content of this blog is unofficial and should be considered advanced configuration advice for anyone who is familiar with Kafka configuration.  I do not recommend changing the default settings in your kafka.properties file without significant research and testing. The kafka.properites file The setting file to be modified is located here on Windows:    C:\Program Files\ArcGIS\Server\GeoEvent\gateway\etc\kafka.properties The original settings in this file were chosen to optimize performance (at the expense of potentially large disk usage).  I recommend you start by adding the log.roll settings at the bottom before you change any of the existing settings. Note: The default settings from Kafka creates a large set consumer offset partitions each with a mimum size of 20 MB. This large number is what gives the system such good performance (parallelism).  A new/empty installation for GeoEvent Gateway requires at least 1 GB of disk space just to get started. Each input/output you add after that will require a minimum of 360 MB additional disk space before you process any events.  Please note all of these sizes are minumum numbers and likely to grow when you use the GeoEvent. You can UPDATE the following settings: log.retention.bytes - This determines a minimum amount of disk space for a single partition.  The default is 100 MB.  This is not a maximum size, and I've seen some partitions grow up to more 3x the setting for this property. For a high velocity data stream, the size of each partition will probably never go below this number.  Also, this is per-partition, so multiply this size by 3 for each input/output you have. If you don't have a lot of high-velocity data, I don't think it will harm anything to reduce this number by 1/2 to 1/3. log.retention.hours - This is the number of hours to leave logs on disk before considering deleting them.  The default is 1 hour.  You can make this shorter by removing this property and replacing it with log.retention.minutes. I have conisdered, but not tested replacing log.retention.hours=1 with log.retention.minutes=30.  log.segment.bytes - This is the maximum size of a log file (the actual file on disk) before Kafka rolls over to another file.  Default is 100MB.  If you have high velocity data, you might end up with a lot of these, if not you might only have one.  I would consider updating this to 50MB, 25MB, or even 10MB. The lower the velocity of your data, the smaller the size you can make this.  The higher velocity your data, the larger this will be. If you set this too small, Kafka will continually be rolling over files. If you set it too big, Kafka will never roll over, and you'll keep log files (and old events in the queue) around forever (see log.roll.ms property below to avoid this). You can ADD the following settings: log.roll.ms - This determines a life span for any specific file on disk; after this many milliseconds, Kafka will roll over to a new file regardless of file size.  I would set this somewhere around 1/2 to 1/10 of your log.retention.hours/log.retention.minutes setting (adjusting for ms).  If you stick to 1/2 and your log files are not rolling over based on size, then you'll rarely have more than 3 log files on disk.  Once again, this totally depends on the velocity of your data.  I would avoid setting this value to anything less than about 5 or 10 minutes (just a gut feeling). log.roll.jitter.ms - this is a fudge factor that allows the sytem to be less punctual when rolling to a new log file. If you are rolling files every 20 minutes and your log file is 20 min old, then Kafka has +- the log.roll.jitter.ms to roll the file over.  My gut says this should be no less than 30 seconds, and generally I would suggest setting it at 1/10 of log.roll.ms (if log.roll.ms=1200000 [20min] then log.roll.jitter.ms=120000 [2min]). Hope this helps! Eric ... View more

Recently, a user asked if GeoEvent supports the "Allow assignment of unique IDs for map service publishing" feature in Map Services.  The short answer to this is YES, but there are some known limitations as described below. Note that these limitations only apply to Map Services and don’t apply to Feature Services.  Feature Services allow you to set unique Layer IDs, but will always present the layers in numerical ordering (ignoring the order you imposed when publishing). NOTE: A workaround for Map Services you don't own/can't control at the bottom of this post. Map Service Layer ID 0 In a map service, if you choose to assign Layer IDs to your layers, you should ensure one of the following: Layer ID 0 IS ALWAYS the First Layer If the first (topmost) layer has Layer ID 0, then the other layers can have any Layer ID as needed.  If you need to re-order layers and the layer with Layer ID 0 is no longer first, it MUST be changed to something other than 0. Do NOT use Layer ID 0 in your map. If the Layer ID 0 is used in any location other than the first (topmost) layer, the GeoEvent Manager user interface may not interpret it correctly.  For this reason, if you use the unique Layer ID feature, it is recommended that you avoid using the Layer ID 0 in your map services.  If you think the order of your layers in a map may change, a best practice would be to avoid the use of Layer ID 0. Changing a Layer’s Layer ID GeoEvent manages its configuration based on the Layer ID. If you change a Map/Feature Service so that the layer’s Layer ID changes, you are responsible for updating your GeoEvent component to use the correct layer.  Some examples are provided below: Example 1: A Map Service that does NOT have Unique IDs enabled with two layers: Wells (0) and Trees (1). A GeoEvent Input polls the map service for the Trees (1) layer. The map service is updated with a new Signs layer added to the bottom of the layer list: Signs (2). The GeoEvent Input does NOT need to be updated: it will continue to work as expected, polling layer Trees (1). There are no errors or issues because the Layer ID of the target layer has not changed.   Example 2: A Map Service that does NOT have Unique IDs enabled with two layers: Wells (0) and Trees (1). A GeoEvent Input polls the map service for the Trees (1) layer. The map service is updated with a new Signs layer that is inserted at the top (so the layers are now: Signs (0), Wells (1), and Trees (2)). The GeoEvent Input MUST be updated: it will look like it is working correctly, poll and create events, but it will be pulling data from the Wells (1) layer, which may or may not coincide with the expected GeoEvent Definitoin for Trees.   Example 3: A Map Service that does have Unique IDs enabled with two layers: Wells (10) and Trees (20). A GeoEvent Input polls the map service for the Trees (20) layer. The map service is updated with a new Signs layer that is inserted at the top and a Layer ID of 5 is assigned (so the layers are now: Signs (5), Wells (10), and Trees (20)). The GeoEvent Input does NOT need to be updated: it will continue to work as expected, polling layer Trees (20). There are no errors or issues because the Layer ID of the target layer has not changed.   Example 4: A Map Service that does have Unique IDs enabled with two layers: Wells (1) and Trees (2). A GeoEvent Input polls the map service for the Trees (2) layer. The map service is updated to add a Signs layer with Layer ID 10, change the Layer ID of Wells to 20 and the Layer ID of Trees to 30 (so the layers are now: Signs (10), Wells (20), and Trees (30)). The GeoEvent Input MUST be updated: it will continue to run without an error indication, polling for the layer Trees (2) but will not get any data. An inspection of the logs will indicate that the ArcGIS Server returned an error “Invalid Parameter”. This indicates the polling request provided an invalid Layer ID of 2, which no longer exists in the service.   Example 5: A Map Service that does have Unique IDs enabled with two layers: Wells (1) and Trees (2). A GeoEvent Input polls the map service for the Trees (2) layer. The map service is updated to add a Signs layer with Layer ID 10, change the Layer ID of Wells to 20 and the Layer ID of Trees to 30 (so the layers are now: Signs (10), Wells (20), and Trees (30)). In GeoEvent the Data Store connection hosting the map service is synchronized to update the service and layer information. The GeoEvent Input MUST be updated: it will be in an error state “”. Since GeoEvent’s cache of the service and layer information has been updated, the input can no longer function properly since Layer ID 2 is not in the cache.   Example 6: A Map Service that does have Unique IDs enabled with two layers: Wells (1) and Trees (2). A GeoEvent Input polls the map service for the Trees (2) layer. The map service is updated to add a Signs layer with Layer ID 10 at the top, without changing the LayerIDs of the other two layers (so the layers are now: Signs (10), Wells (1), and Trees (2)). In GeoEvent the Data Store connection hosting the map service is synchronized to update the service and layer information. The GeoEvent Input does NOT need to be updated: it will continue to work as expected, polling layer Trees (2). There are no errors or issues because the Layer ID of the target layer has not changed in the cache nor the service.   Example 7: A Map Service that does have Unique IDs enabled with two layers (from top to bottom): Trees (1), and Signs (0). A GeoEvent Input polls is configured to poll the map service for the Signs (0) layer. However the GeoEvent Manger user interface may not allow the user to select the Signs (0) layer (it will always select the Trees (1) layer). Workaround If you find a need to use a map service that has the Layer Id 0 in a location other than the first (topmost) spot, you can use the Poll an External Website for JSON input and hardcode the layer ID in the URL you provide. ... View more