Latest Contributions by RJSunderman

Hello Drew -- What you want to do is not possible using only a Filter and Output when writing data out to a JSON or CSV (delimited text) file. Event records received and adapted by an input should be considered atomic. In other words, a GeoEvent Service (generally) doesn't cache or retain any information about event records previously processed. You cannot compare one event record to another, so there is no way to determine that an event record whose attribute Name matches 'John' is similar to another event record -- at least not using a Filter. The Event Volume Controller processor does cache data in order to emit only one one record matching some filterable criteria every "so many seconds". This doesn't do exactly what you want -- to emit exactly one event record and discard all duplicates. I mention it only to highlight that you need something that caches and holds event records in order to compare one to another. I think what you probably want to do is use data from event records being processed to update a feature record, via a feature service, in a geodatabase. You can choose to update just the feature record's date/time attribute, using the attribute field Name as the TRACK_ID to determine which "user" feature record to update. I would probably choose to record the date/time the data was last observed and the feature record updated. You can then script a query to pull data from the feature service and create the "logged once only" list of data records with specific user names. However, since GeoEvent Server (generally) receives, adapts, and processes every data record independently, every data record matching a Filter you configure will be routed through whatever workflow your GeoEvent Service is configured to perform and direct all processed event records to an Output. If that output writes data to a system file, every data record with a Name matching 'John' will be written to the file. If that output updates a database record, every data record with a Name matching 'John' will be used to update a record in the database.  Hope this information helps -- RJ ... View more

Hey Jamie -- ... those two fields are updated in the hosted feature [but] all other fields are overwritten to null. I don't understand why fields not mapped are being updated. Is it even possible to update one field and leave the rest untouched? Is the Field Enricher (Feature Service) processor required? I'm not sure the question above was ever answered, so as a follow-up: You are correct when you said, earlier, that the purpose of a Field Mapper is to copy data from one data structure / schema / GeoEvent Definition to another. You generally use a Field Mapper to simplify and/or flatten a data structure to match the schema a Feature Service expects. That way, when an outbound connector (or "Output") deconstructs an event record to form a REST request, it can send the request to a feature service and the data in the client request will conform to the feature service's requirements. It is possible to update a single feature record attribute (or just a few attributes) and leave others untouched. I would refer you to the article Using a partial GeoEvent Definition to update feature records for details on how to do this. Bottom line up front:  If you leave fields unmapped in a Field Mapper configuration you are actually writing null values into the target data structure. So, yes, the null values end up being used to update feature record attributes overwriting whatever data the feature record had. A Field Enricher (Feature Service) processor does something very different from a Field Mapper. The latter copies data from an event record to organize existing data in a different data structure. A Field Enricher performs an attribute join between an event data record and an external data record to incorporate data not present in an event record (enriching the event record with related data). Hope some of this helps clarify tasks the processors you are configuring are actually doing. -- RJ ... View more

@JamieLambert -- Have you taken a look at the blog Debug Techniques - Add/Update Feature Outputs? If you request DEBUG logging on the com.esri.ges.transport.featureService.FeatureServiceOutboundTransport component logger you can see the REST requests your configured Update a Feature output is sending to the ArcGIS Server feature service. The DEBUG logs in GeoEvent Server give a lot of information you can use to figure out what's missing from the REST requests. The logging was reworked for the 10.8.1 release with even more information provided via TRACE level logging, if you need the component logger to be most verbose. You can also work with the updateFeatures endpoint for the feature service in the ArcGIS REST Services Directory (taking GeoEvent Server out of the equation for now, while debugging) to iterate and try different requests to see what a web client needs to send to be able to successfully update feature records. ... View more

@NathanKoski -- I believe the issue you are running into is related to a couple of bugs specific to using GeoTagger processors configured in parallel as you show in your illustration. A fix was included in the latest 10.8.1 patch which addresses one of the issues (when writing enrichment / tagged values to existing fields). That issue has also been addressed in the 10.9 / 10.9.1 release. If you try to use a pair of GeoTagger processors working in parallel to enrich event records as tracked assets enter and leave geofenced areas -- and configure both processors to write the enrichment / tagged values to new fields however -- there is a risk that the creation of new GeoEvent Definitions will cause an exception which in turn causes a loss of connection between GeoEvent Server and the GeoEvent Gateway. The only recovery if this happens is to stop and restart the GeoEvent Gateway service. This issue has not yet been addressed. Two suggestions to work around the problem are: Apply the latest 10.8.1 patch or upgrade to the 10.9.1 release. Add a field to a GeoEvent Definition you own/author and then configure the GeoTagger processor(s) to write their values to an existing field. You will need to use a Field Mapper in front of the GeoTagger to prepare the event record schema so that records routed to the GeoTagger have the empty/null field in their schema which the GeoTagger processor can write to. Do not configure two GeoTagger processors to work in parallel. Configure separate GeoEvent Services each with a single GeoTagger processor and allow that single processor to create a new GeoEvent Definition as needed when writing to new attribute fields. The first is the preferred option. If you continue to have problems, please open an incident with Esri Technical Support and refer to the following two bug reports:  BUG-000142170, and BUG-000132928   @DanWade   @RichardsonAluvilayil  ... View more

A new document has been added to the GeoEvent Server Documents board which illustrates how to add new attribute fields to a BDS Data Source’s map or feature service schema. This can be done using either the Enterprise portal content item manager or ArcGIS Pro to update the schema of an existing hosted feature layer. Please note that while new attribute fields can be added to an existing data source you are not currently able to remove or delete attribute fields from an existing hosted feature layer when data is stored within a spatiotemporal big data store. Click to jump to the article on the GeoEvent Server Documents board.   @EricIronside   @yujingwu1  ... View more

Hello Walter -- Up front, I'm going to ask that you open an incident with Esri Tech Support to get a specialist assigned who can help reproduce the issue on our end. This is for traceability in case the behavior you are seeing is a bug which needs to be addressed by the development team (as opposed to something we can configure in your deployment or environment that solves the issue for you). A couple of things I know about stream services:  There is only one web socket running for each stream service output. This web socket is run as part of the GeoEvent Server's JVM. When multiple clients subscribe concurrently to the web socket they all suffer a deterioration in performance. I do not know what the upper limit is on how many clients can concurrently subscribe to a single stream service - only that there is little you can do to scale out the stream service for many hundreds or thousands of subscribers. You cannot, for example, configure more than one web socket resource to run and support a stream service you expect to experience a high volume of subscribers. There are four component loggers for stream services which you might try setting to log DEBUG messages to see if you can capture an error or warning around the time the stream service stops broadcasting data. com.esri.ges.datastore.agsconnection.NewStreamServiceBuilder com.esri.ges.framework.streamservices.client.StreamServiceClientImpl com.esri.ges.framework.streamservices.client.AGSConnectionStatusListenerStreamServiceClient com.esri.ges.transport.streamService.StreamServiceOutboundTransport I do not know off hand how verbose these component loggers are relative to data ingest volume / velocity. If you request DEBUG logging, monitor your karaf.log found beneath ...\ArcGIS\Server\GeoEvent\data\log to gauge how fast the log file is growing and log file rollover. If the logging is too verbose you could easily miss the data you are trying to capture as the current karaf.log fills-up, rolls over, and older rolled log files are deleted. As one stream service appears to be working reliability and the other seems to be failing every day or so, is there anything obviously different between the two?  Are both stream services broadcasting roughly the same number of Point (my assumption ?) feature records per second?  Do both stream services have roughly the same number of concurrent subscribers?  You indicate that the problem does not manifest when the same stream service is run on a second machine, but then, there isn't anyone subscribing to that instance of the stream service. I know that Esri Tech Support has a similar case / incident they are helping the development team reproduce -- not specifically stream service related, but related to event processing stopping after a variable length of time (reportedly 2 - 5 days). To briefly describe some underlying GeoEvent Server architecture for you, there are two Kafka topic queues working with each GeoEvent Service. The first routes event records from an input (the producer) to a GeoEvent Service for processing. The second routes processed event records from the GeoEvent Service to an output (a consumer). If something were to go wrong with the Kafka event broker I might expect an input would continue to ingest and adapt data to create event records, but those event records would not be processed by a GeoEvent Service -- you would see the event count of the input increase as data was received, but the 'In' / 'Out' event count of the GeoEvent Service in which that input was incorporated would not increment. Or, if the second Kafka topic had failed, you might see event data being processed and the 'Out' event count for a GeoEvent Service incrementing, but the stream service output's event count would not increment. Knowing which of these you are observing (if either applies) will help Esri Technical Support better address an incident you submit with them. To help Esri Technical Support we will need you to capture a sample of the radio device or iPad device data being sent to GeoEvent Server. It doesn't look like there is anything "stateful" in your GeoEvent Service (e.g. I don't see an Incident Detector or any particular filtering logic) which might prevent us from running the same few hundred event records in a loop through GeoEvent Server for 48 hours. If you have reason to believe that we need more temporally consistent / contiguous data, we'll need your help to get a snapshot of that so that we can run the system for 48 hours to see if we can recreate the failure in-house. Please also create an XML snapshot of your configuration using the 'selective' export option to export just the GeoEvent Service whose stream service output is having a problem. This will simplify the XML snapshot by not including inputs, outputs, GeoEvent Definitions, or other configurable elements not shown in the relatively simple GeoEvent Service you illustrated above. Hope this information helps -- RJ  ... View more

Hey Adam -- Thank you for highlighting these new server-side (datesInUnknownTimeZone) and client-side (timeReferenceUnknownClient) properties. Checking with the development team, I was told that service publishers, beginning with the 10.9 release, can now set datesInUnknownTimeZone to TRUE when publishing a feature service (the default for this property is FALSE). If an application sends a query, addFeatures, or updateFeatures request and does not explicitly set the timeReferenceUnknownClient query parameter to TRUE in the request -- and the service was published with its datesInUnknownTimeZone parameter set TRUE -- the client will receive an HTTP error code response. This is to protect client applications querying newer 10.9(+) services which make use of the new specification from receiving unexpected data. Services you publish using GeoEvent Manager at 10.9 / 10.9.1 will continue to use the default setting of FALSE for the datesInUnknownTimeZone parameter. ArcGIS Server feature services and Enterprise portal hosted feature layers will not change the default for this property when services are upgraded from 10.8.x to the 10.9.x release. Therefore, applications sending client queries should expect to receive the same epoch long integer values from feature services and hosted feature layers they have been receiving regardless of whether the services are (pre-upgrade) 10.8.x services or have been upgraded to the 10.9.x release. I do not have a 10.9.1 release candidate deployment I can use to check right now, so I would ask that you "trust but verify" what I've said above. You can examine an ArcGIS Server feature service's (or an Enterprise portal hosted feature layer service's) JSON specification by navigating to the service in the ArcGIS REST Services Directory and clicking the JSON link in the upper-left corner of the service's web page. I would look to see if the upgraded service's JSON specification has the datesInUnknownTimeZone parameter set TRUE or kept the FALSE default. I would then check that web applications (e.g web maps) you have configured display the expected date/time when looking at a feature record's attributes in a pop-up or table. I would also configure a GeoEvent Server input to query the ArcGIS Server feature service and/or Enterprise portal hosted feature layer you have upgraded to retrieve feature records and write those out to a JSON file. Use the https://www.epochconverter.com on-line utility to convert the epoch long integer GeoEvent Server has written out to its JSON data file to confirm the queried values. If you have any trouble with this, please open an incident with Esri Technical Support. (I will be running these same tests myself as I was previously unaware of the new service property and corresponding client query parameter.) -- RJ ... View more

Hey Dean -- What you indicate regarding copying a GeoEvent Definition which has been generated for you to apply tags or event to adjust data types is the better practice. I often have to adjust a GeoEvent Definition to specify that a numeric value be handled as a Long, or even as a Date when receiving epoch values, rather than allowing the more generic numeric type of Double data type to be used. Considering the application of a TRACK_ID tag, event definitions generated for you by an inbound connector or processor cannot accurately guess which field might hold a unique record identifier, so the TRACK_ID tag is not applied when generating a new GeoEvent Definition. I've also seen what you mention with duplicate GeoEvent Definitions being created especially when an XML snapshot of a configuration is loaded and what I refer to as "managed GeoEvent Definitions" are part of the snapshot. Frequently an input configured to create a GeoEvent Definition will create a new GeoEvent Definition when receiving data for the first time following an administrative reset and configuration import. The "duplicate" event definition created has the same schema and even the same name as the event definition captured by the snapshot. But the underlying GUID for the two event definitions is different which can lead to other problems. The less you allow components you configure to create GeoEvent Definitions for you on-the-fly the more distinct and specific your configuration becomes -- which tends to improve resiliency and overall solution reliability. Auto-generated GeoEvent Definitions can be useful, but should always be replaced with copies you have tailored specifically for your configuration and operations. Auto-generated GeoEvent Definitions are really just best guesses for an event record's data structure and the guesses are generally good, but rarely 100% matches of what you want. -- RJ ... View more

@EricSpangenberg -- Apologies that your enquiry went so long without a response. If you have not already, please open a support incident with Esri Technical Support for this issue. It is not feasible, and in fact can be quite dangerous from a security perspective, to attempt an in-place upgrade of a third-party component GeoEvent Server incorporates as part of its Karaf application container. Please indicate to Esri Technical Support the release of ArcGIS Enterprise and GeoEvent Server you are working with. If this is a vulnerability not covered by the ArcGIS GeoEvent Server Security Update 2021 Patch we need to evaluate this with our internal security team. If you are working with a release prior to 10.6.1 we probably need to discuss with you the possibly of upgrading to an ArcGIS Enterprise release for which an appropriate security patch is available. Either way, we need a support incident opened with Esri Technical Support for traceability. Thanks -- RJ ... View more

Hello Dean -- My apologies that your inquiry has not received any responses.  Have you opened an ticket on this with Esri Technical Support?  I would like to have these sort of issues tracked.  While I have not had a problem configuring Incident Detector in deployment I have set up for regression testing, I did have someone reach out to me that was having a problem with their deployment and configuration which I was unable to reproduce. If this is still an issue for you, and you are so inclined, please work with Esri Technical Support to try and establish a reproducible case. ... View more

William -- The product development approach for GeoEvent Server avoids developing connectors for specific devices or platforms. While there are some samples on the GeoEvent Server Gallery for data provider or device specific inbound connectors (e.g. Verizon Connect Connector for GeoEvent Server, or Geotab for GeoEvent Server), when considering connectors available out-of-the-box you generally want to identify what transport protocol will be used to send data to GeoEvent Server (e.g. HTTP, TCP, UDP, WebSocket) and how the data records will be formatted (e.g. generic JSON, generic XML, delimited text). The answer to these two questions will guide you toward a configurable out-of-the-box inbound connector. Since you mention the ActiveMQ connector, I'll offer that over time we have exposed (as SDK samples) connectors based on "message queues" and "message brokers" such as ActiveMQ, RabbitMQ, MQTT, and Kafka. Not all of the samples on the Gallery are plug-and-play with the latest release of GeoEvent Server. Connectors based on different message brokers follow GeoEvent Server's evolution. We started with ActiveMQ, migrated to RabbitMQ, and are now using Kafka internally to queue and manage event records. None of this changes GeoEvent Server integration fundamentals of identifying a data transport protocol and data record format. I am not aware of anything developed specifically for WebLogic Thin T3 Client. This could be something you develop a specific connector for using the GeoEvent Server Java SDK. Implementation support for this type of project is available through Esri Professional Services. Hope this information helps -- RJ ... View more