Latest Contributions by RJSunderman

Hello Nathan - I am not familiar enough with ArcGIS Collector to comment on the behavior you are observing that OBJECTID do not increment along with the track points in the temporal order they were received. You're in the right GeoNet Subspace for questions on the application ... so I'll let someone from that team comment on whether or not what you're observing is considered a bug. From a GeoEvent‌ perspective, I can tell you that https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2015/08/20/polling-feature-services-for-incremental-updates?sr=search&searchId=6e967e3e-5cd3-4dfd-a328-36704d6055d4&searchIndex=1‌ can be problematic when using OBJECTID as the key value for determining when new feature records have been added to the feature service you are polling for input. Most often we see the problem when multiple contributors are adding feature records to the geodatabase feature class. ArcGIS Server has a mitigation strategy which allocates OBJECTID in blocks of (I believe) 400 to each contributor to avoid race conditions in which two contributors ask for the next available OBJECTID, both are told that the ID 'X' is the next in line, and both attempt to create a new feature with the same OBJECTID. What you'll see is feature records being created with OBJECTID values 1, 2, 3, 4 .... then the ID values will appear to jump to 401, 402, 403, ... and maybe later to 801, 802, 803, ... The problem with this from a GeoEvent Server perspective is that the GeoEvent input is querying for feature records using the greatest-value of all the OBJECTID values observed in the last feature record set queried from the service. This means that once feature records with the 80x IDs start being created, the input will remember seeing values in the eight hundred range and will construct a query with a WHERE clause that excludes any feature records whose OBJECTID value is less than 803, 804, 805, ... Then, when one of the other contributors operating within the block of OBJECTID values allocated to his/her operations, creates new feature records 6, 7, 8, 9, ... none of these are polled by the GeoEvent input because it is using a key value in the 800 range when querying for "new" feature records based on OBJECTID. You'll be much better-off using a date/time field when querying using the GeoEvent input's "incremental update" capability. But even then, if ArcGIS Collector is adding feature records out-of-order, or is adding them as a batch without setting the date/time each track point was actually collected, you'll have trouble using the GeoEvent Server's polling input to identify feature records which are both "new" and get them in the order they were originally collected. Also remember that the geodatabase only honors date/time resolution to the second, so even if ArcGIS Collector specifies sub-second / millisecond precision, that precision will be lost. The date/time values of the persisted feature records will either be truncated or rounded to the nearest whole second (I don't remember which). I recommend taking a look at the blog I referenced above. Depending on how you've chosen to disseminate the event records you've processed, polling for incremental updates can cause you other problems when the GeoEvent Server service is restarted - which will happen if an administrator restarts the ArcGIS Server service or when the server is rebooted.  Hope this information is helpful - RJ ... View more

Vladimir Strinski‌ The illustration you included in your post from 11-Oct-2017 is part of an unrelated issue introduced in the 10.6 release. GeoEvent Definitions which were not suitable for use when publishing a 'Store Latest' feature service (or any feature service GeoEvent Server would use to add/update features) was triggering a sticky validation message -- by which I mean even if you selected a different GeoEvent Definition, one that was suitable and had a field tagged TRACK_ID, the GeoEvent Manager web application's panel still displayed the error message that GeoEvent Definition <name> is missing a TRACK_ID tag. Esri Support was tracking this as issue BUG-000113943 ... which we've also addressed as part of the 10.6.1 product release. - RJ ... View more

Variants of this issue were documented by Esri Tech Support under the following: BUG-000107723 BUG-000111776 BUG-000114252 Testing all reported behaviors, it appears that these issues have been addressed in the 10.6.1 release candidate. ... View more

Hello Sam - No, I don't think so ... but I'm not as familiar with DEM as I am with simple feature classes. GeoEvent Server offers two ways to enrich event records received in real-time: 1) GeoTagger - This is essentially spatial enrichment 2) Field Enricher - Essentially an attribute enrichment Having established a feature set, you can import the feature geometries as geofences. You can then use a GeoTagger to determine if the Geometry associated with an event shares a spatial relationship with one or more geofences (e.g. intersects, within, crosses, disjoint ...) Event enrichment using a GeoTagger is limited to enriching the event record with the name of the geofence with which it shares the specified spatial relationship. If you wanted to enrich the event record further, for example, to bring other attributes from the feature record whose geometry was the original source of the geofence into the event record ... you could use the geofence's name as the primary key for an attribute join and then use a Field Enricher to bring in the feature record's other attributes. Normal field enrichment is attribute only. Your event record's attributes are used to create a virtual table. You specify which event attribute should be used as the primary key, which feature record attribute should be used as the foreign key, and which feature attributes you want to bring into the event record as part of the enrichment. For your use case, you would have to first establish a fishnet of polygons whose center point was the average elevation for the area. Then you could GeoTag to determine which grid cell an incoming point was within, and Field Enrich to obtain the elevation information from the feature record(s) used to establish the fishnet of geofences. If this is not practical, because you have a wide area of interest with significant elevation variation across the DEM and/or very small elevation grid cells, then you would probably have to use the GeoEvent Java SDK to develop a custom spatial join which could work directly with the DEM without having to first establish a feature set of polygons representing a fishnet over yout DEM. - RJ ... View more

Hello Annina‌ For the current 10.6 and previous 10.5.x releases I would recommend the following videos, available from the Esri YouTube Channel: Creating Processors using the GeoEvent SDK Developing Real-Time Web Apps with the ArcGIS API for JavaScript Most likely you are going to need to develop a custom processor for GeoEvent Server, using the Java SDK, to handle decoding of the encrypted hex attribute values. The GeoEvent Server's Field Calculator is not at all the same as the Desktop / ArcMap Field Calculator. GeoEvent's processor supports some limited string manipulation using methods from the Java String class. You can read more about these in the on-line documentation (updated to include examples for the 10.6 release):  Processors - Field Calculator Unless / Until we develop a Field Calculator that accepts Python Script - you're limited to the above. -- RJ ... View more

I don't know that we have a work item in the GeoEvent Server's backlog to create a processor which will allow you to enter Python Script to execute against an event record received for processing by a GeoEvent Service. However, there is work being done on the realtime module within the ArcGIS API for Python‌ which might provide functionality you are looking for. To folks voting-up this idea ... would you please comment with some additional detail on how you would like to use Python with GeoEvent Server?  Are you looking to write python script to calculate values like the ArcGIS Desktop / ArcMap "Field Calculator" (vs. the GeoEvent Server "Field Calculator" whose expression interpreter does not support script/code)? Or are you more interested in scripting system administration tasks, like those you perform from within GeoEvent Server's Manager web application? Are you perhaps looking for ArcGIS API for Python to expose endpoints from the GeoEvent Server REST API? - RJ Shannon Kalisky‌ ... View more

David Martin‌ - I think your summary / take-away is accurate. A load balancer could be used to simply round-robin event records to available machines because, yes, the inputs on all the machines are configured the same. Doing this may not be optimal as the Kafka event distribution at 10.6 will still attempt to do *its* job and distribute event records with a given TRACK_ID to a single server ... with fail-over to another server only if the primary goes off-line. To clarify, as long as all machines remain up and running, the fact that "state" is not shared between machines does not matter. Kafka is handling event distribution according to TRACK_ID and is sending event records with a given TRACK_ID to a consistent server. It is only when a machine fails and Kafka has to choose a another machine to process event records with that TRACK_ID that event records previously handled by the machine that failed may produce false positive analytics ... since the secondary machine (which has never seen event records with *these* TRACK_ID values before) has to begin to accumulate a history of the event records it is now receiving. The point I was trying to make is that -- if you elect to handle event distribution yourself, using a round-robin approach across multiple independent "silo'd" instances -- Track Gap Detection, Incident Detection, and spatial operations such as ENTER and EXIT are not going to work because event records with a given TRACK_ID are not being routed to a consistent machine. Also, the real purpose of the load balancer is not traditional load balancing. The purpose in this case would be to expose a single receiving endpoint (the URL of the LB) to data providers and obscure (from data providers) the fact that multiple endpoints are available. This potentially provides your solution some resiliency when a given machine fails ... ingest does not come to a screeching halt when data providers continue trying to send data to the failed machine's endpoint(s). Your LB, through a "sticky session", would choose which machine should act as the primary ingest node for a given inbound data stream and fail-over to a secondary machine only if the first machine became unavailable. - RJ ... View more

>> Is there a scheduled release date (or approximate ETA) for the 10.6 patch?   We are working now to determine what will be committed for a 10.6.1 final release, what portion of that will be ported back for 10.6 Patch 1 … and if anything critical must be released as part of the patch which cannot be committed to 10.6.1 (not a configuration we desire, but sometimes have to do).   I would look for 10.6 Patch 1 to be released by UC in July, but it may come out earlier, as soon as 10.6.1 is handed off to the Esri Release Team for final certification and testing.   >> In a multiple-machine site (minimum 3 nodes), should we expect to have to license each and every node with a GeoEvent Server license? (i.e. pay for 3 licenses)   Yes, every deployment / installation of GeoEvent Server must be licensed using (at least) the GeoEvent Server license for that server role. Optionally you can also apply an ArcGIS Server standard / advanced license to the machine you’ve deployed GeoEvent Server, if you want enable the full capability of the ArcGIS Server on that machine – but it is sufficient to license both the ArcGIS Server and GeoEvent Server using a GeoEvent Server license role if you only want the real-time capabilities on that machine.   The same, by the way, is not true for ArcGIS Data Store. With a single ArcGIS Enterprise license you can install ArcGIS Data Store on as many machines as you like, configuring each as a node of the spatiotemporal big data store registered with the Portal’s hosting GIS Server, without having to purchase additional licenses for the instances of ArcGIS Data Store you are deploying.   >> I take it from this that a multiple-machine site exposes no single endpoint for receipt of an http request (whether POST or GET)? If that's the case, then I guess a suitable load balancer needs to be added up-front which can health-check each machine in order to distribute such requests only to healthy ones?   That is correct. Nothing related to the GeoEvent Gateway, GeoEvent Server, or ArcGIS Server components provide a single endpoint (in a multiple-machine architecture) to which a data provider can direct data when actively pushing data to GeoEvent Server. Yes, you can probably configure a load balancer with a “sticky session” to establish affinity with a particular instance of GeoEvent Server, using one of the monitoring endpoints in the GeoEvent Server Admin API to determine if that instance is still available. But I’m not the one to advise you on how to do that – I’ve no experience in that area of system architecture.   >> Which could just as easily be done with a suite of single-machine sites, for inputs of this nature, provided no gap detection, geo-fencing or similar is required?   Yes, as I understand load balancing technology, you could use a LB to route event records to a single-machine “silo’d” instance of GeoEvent Server just as easily as you could an instance participating in a multi-machine configuration. The drawback would be that the single machine instance would not have any peers to which it could further distribute event records for real-time event processing. Also, as you note, if the LB did not guarantee that all tracked assets (those with a given TRACK_ID) were routed to the same machine, real-time analytics like Track Gap Detection, Incident Detection, and spatial operations such as ENTER and EXIT will not work consistently as one machine's "state" (i.e. knowledge of a prior event record) is not propagated or shared between machines in a multi-machine site. Geofencing would work, as long as you had explicitly loaded a consistent set of geofences to each GeoEvent Server instance (in a single-machine "silo'd" configuration).   >> For "guaranteed" receipt, the onus would then just remain on the sender (or middleware) to re-send if no 200 OK response is received from GeoEvent Server?   No, unfortunately that won’t work. An HTTP / 200 response does not imply success in constructing or handling of an event record.   Every inbound connector has two components:  an adapter and a transport. It is the transport’s responsibility to receive a payload (think “raw bytes”) and confirm with the data provider that the data was successfully received.  That’s all the HTTP / 200 response means.   It is the adapter’s job to reference a schema (e.g. GeoEvent Definition) and construct an event record from the payload provided by its associated transport. That’s why you can configure new connectors to, say, receive JSON over TCP – when out-of-the-box there is only a receive delimited Text over TCP – without writing any code. You are just configuring a new inbound connector paring an out-of-the-box transport with an adapter.   The HTTP / 200 returned from an inbound connector, by its transport, does not mean that the inbound connector’s adapter was successfully able to instantiate an event record for processing, that the event record was successfully processed by a GeoEvent Service (configuration errors in a processor might end up throwing exceptions) or that an output was able to disseminate the event record’s data. - RJ ... View more

Hello David –   >> However, what about solution AVAILABILITY, which could manifest as a requirement even when consuming very low message volumes, but where the business criticality of the solution is high? I assume that this is best met using a multiple-machine site (once patched)?   Perhaps … there are multiple considerations which is why I recommend folks who are looking at scaling out real-time solutions work with their Esri Technical Advisor or contract with Esri Professional Services for consultation. I’ll try to answer you briefly here however.   Depending on the type of input you are using, a multiple-machine architecture might provide the failover you are looking for. Our testing when using an input which polls an external source for data was very positive. If one machine in a multi-machine site were to go off-line we’ve confirmed that another machine will “adopt” and being running the input so that event ingest doesn’t terminate with the one machine.   If you’re using an input like a JSON REST receiver, or the TCP/Text input, to push data to GeoEvent, then the burden is on the data provider to specify which machine, endpoint, port, (etc.) to push the data to. The challenge here is that there is no built-in way to signal an external data provider that a particular GeoEvent Server instance has gone off-line, so it is difficult to know when to direct data to a different machine or endpoint. In this case a multi-machine setup is probably not going to help; a significant amount of custom development would be required on your part as a solution architect.   I might recommend, for the case where you have low event velocity / volume but message data is critical, that you set up multiple independent GeoEvent Server instances to provide basic redundancy. Process the critical inbound data in parallel across multiple redundant instances and write the resulting feature records out to parallel feature services. The challenge, then, is advising consuming clients when to use the primary feature layer and when to switch to a failover.   Please keep in mind that GeoEvent Server, at a fundamental level, was architected with the assumption that data from sensor networks would be both frequent and periodic. If a reported temperature reading is missed or dropped it’s not a big deal because another will come along from the sensor momentarily.   This assumption is not entirely compatible with stipulations that every event record is critical, neither frequent nor periodic, and data loss cannot be tolerated. If GeoEvent Server receives the data the system will reliability process and disseminate event and feature records. Network stability, datababase failure, feature service availability and recycling … there are many different points at which a system which distributes capability across multiple components might result in a loss of data. System architecture and solution design can address these, but I do not want to take this discussion into one of general “high availability” which often means different things to different customers in different contexts.   >> There appears to be a recommendation, however, that multiple-machine sites should have a minimum of 3 GeoEvent Servers. This could feel a bit excessive where message volumes are low. Is there a good reason why a multiple-machine site can't just have 2 GeoEvent Servers?   Yes, and it has more to do with general computer science principals than anything specific to GeoEvent Server. If two nodes are receiving input and one goes off-line, when it comes back there is no majority or quorum to stipulate that the node coming back on-line is out-of-date. You are left with two peers, each thinking they have the latest state. When you have three (or more) nodes and one fails, upon recovery the other two can establish for the node coming back on-line what “latest” looks like. So, the recommendation for three GeoEvent Server instances is to avoid a “split-brain” condition.   >> Another caveat with a multiple-machine site appears to be that outputs cannot include Stream Services. As such, there will be a need to store geoevents prior to displaying them in a client, and the client will need to poll a feature service to retrieve them. Is that assumption correct?   No, you should be able to leverage stream services in either a single-machine or a multi-machine deployment.   We did find a bug in 10.6 where consumers subscribing to web socket endpoints exposed by machines in a multiple machine deployment would sometimes not receive any feature records. The reason was that the web socket they subscribed to was not actually broadcasting any data. If the client were a web map, for example, a few manual “F5” refresh requests would eventually have the client subscribe via the server instance that actually published the stream service, which was the only one whose web socket was actually broadcasting feature records.   We’ve addressed that issue and the “primary” stream service is now using the message bus to “fan out” and send feature records to the other stream service instances so that a client can subscribe to any machine’s web socket and get all of the feature records processed across the multiple machines – regardless of which machine received an event record and which machine actually processed the event record. This fix will be included in our first patch for the 10.6 release.   >> Do you have any recommendations as to when to consider deployment of the spatiotemporal data store? (e.g. a threshold level of throughput).   To quote an associate, you know you’re dealing with big data when your traditional methods and tools begin to fail. Many folks successfully leverage feature services backed by a traditional relational database (RDBMS) and use the Portal’s hosting server’s managed database to persist their feature records.   Where reliance on an RDBMS begins to fail is when you need to persist more than about 200 events each second. There is a lot of overhead to the REST requests GeoEvent Server has to make on a feature service’s add (or update) feature endpoint to initiate a transaction with a relational data store. Depending on network and database tuning you may discover that transactional latency allows you to persist fewer event records each second. This is the primary reason to consider switching and being using the spatiotemporal big data store.   GeoEvent Server discovers, through Portal and the Portal hosting server, credentials needed to open a socket connection directly to Elasticsearch (the no-SQL database behind the spatiotemporal big data store). Avoiding REST requests means that feature records can be persisted at much higher velocity into the database.   A second reason to consider using the spatiotemporal big data store is that Elasticsearch natively provides data replication across multiple nodes when you establish a data ring by installing ArcGIS Data Store (and selecting the spatiotemporal big data store capability) on multiple machines.   If you include three nodes of the spatiotemporal big data store in your architecture – again, trying to avoid the potential of a split-brain condition – and one of your database server machines were to fail, only one node of the data ring would go off-line. Client requests for data will be redirected to surviving instances. Replicas of your feature records, created automatically and managed by Elasticsearch, will be returned to requesting clients.   Hope this information helps – RJ Josh Joyner‌ ... View more

Something to consider if you want to more precisely specify RAM allocation for the JVM is to include both the maximum and the minimum allocation. The Java heap size is controlled using the Xms and Xmx switches. Xms is the starting heap size; Xmx is the maximumum heap size. You can read more about this and why you might want to consider the strategy in the following article ... https://codecurmudgeon.com/wp/2012/05/java-memory-settings-jvm-heap-size/ ... View more

Thank you [email protected]‌ - I've repaired the links. - RJ ... View more

tshippee-esristaff‌ Good collection of information on GeoEvent geofences you can reference in class ... ... View more

I normally prefer exporting my GeoEvent configuration to an XML file, then uninstalling GeoEvent Server and installing the newer release (vs. following an upgrade-in-place workflow). Product upgrades make more sense for ArcGIS Server and Portal for ArcGIS where exporting and importing your site configuration can be more difficult. For GeoEvent Server, it's really easy to snapshot the configuration as XML and then selectively import just the inputs, outputs, GeoEvent Services (etc.) that you know you're using - or import the entire configuration. In the case you're describing, where you want to scale your deployment from a single-machine architecture to a multi-machine architecture ... I would definitely recommend you export the GeoEvent configuration, perform whatever s/w upgrades are needed to move to 10.6, establish your ArcGIS for Server site with its multiple machines and multiple instances of GeoEvent Server, then import your GeoEvent configuration once, from the XML. I would then test to make sure that all GeoEvent Server instances 1) receive the configured elements; and 2) event processing is distributed across the machines in the site when event data is sent to any one machine's input. - RJ ... View more