topic Re: GeoEvent and High Availibility in High Availability and Disaster Recovery Questions

GeoEvent and High Availibility

AzinSharaf — Tue, 25 Jul 2017 22:47:05 GMT

This would be our Portal and ArcGIS Server high availability deployment.

We want to add a HA GeoEvent Server to this architecture. What would be the best solution based on the following limitations/recommendations?

1- Don't use multi-machine sites with GeoEvent

2- Don't install GeoEvent in your base GIS site

If we create two independent GeoEvent site, we should sync them manually? Any better solution?

Re: GeoEvent and High Availibility

RJSunderman — Wed, 26 Jul 2017 23:39:08 GMT

Hello Azin -

The ArcGIS Server documentation includes reference architectures for deploying additional server roles (such as GeoEvent Server) to your ArcGIS Enterprise. The terms I'm using here are consistent with the 10.5.x product release. There are reference architectures for high availability as well as for GeoEvent Server.

Based on the icons you included in your system architecture illustration, I might assume you've seen these references - but I wanted to make sure.

I would point out that the reference architecture for GeoEvent Server which depicts a horizontal scale-out deliberately has white space between the machines:

The white space is intended to depict your first stipulation -- that you do not 'Join Existing Site' when installing the GIS Server component of ArcGIS Server with the intention of later installing GeoEvent Server. Each of the three GeoEvent Server machines depicted above exist in their own site. They do not know anything about one another. This is to prevent leveraging a concept we refer to as "GeoEvent Clustering" which we introduced with the 10.3 release. Changes to address some issues were included with the 10.4 release. We eventually realized that the concept was fundamentally not working as we intended, and the advice now for releases 10.3 / 10.4 / 10.5 is to not have more than one GIS Server in a site when GeoEvent Server is installed.

We are working on developing a "Gateway" component for 10.6 with the intent that GeoEvent Server will again offer some resiliency and improved scalability. However, until then, your challenge is to figure out how to configure an external component to handle event distribution to two or more instances of GeoEvent Server which do not know anything about one another.

We prepared a GeoEvent Server Resiliency tutorial which proposes using Kafka as an external message broker. Kafka provides load balancing to your real-time event message ingest by holding messages in a queue until consumers (individual GeoEvent Server instances) fetch them at the rate they are able to process them. My terminology may be off; I would recommend you refer to Kafka's documentation as you read through the GeoEvent Server Resiliency tutorial.

I hope this information is helpful --

Re: GeoEvent and High Availibility

AzinSharaf — Fri, 28 Jul 2017 22:27:29 GMT

Thanks rsunderman-esristaff

Re: GeoEvent and High Availibility

HarroldSompotan — Mon, 22 Jan 2018 23:56:47 GMT

Hi RJ,

Now that Arcgis for Server and Geoevent Server 10.6 is available, and the Gateway components are in place, are we still encouraging other to go with a siloed Arcgis Geoevent Server instead of Arcgis Geoevent Server in a multimachine site?

Currently the documentation at 10.6 still mentions below:

"It is NOT recommended to use multi-machine sites with ArcGIS GeoEvent Server. Instead, you should configure multiple single-machine sites to dedicate ArcGIS GeoEvent Server machine resources to specific real-time data streams:"

Deployment patterns for ArcGIS Enterprise—ArcGIS Enterprise | ArcGIS Enterprise

Re: GeoEvent and High Availibility

RJSunderman — Tue, 23 Jan 2018 17:56:31 GMT

Yes, with the introduction of the GeoEvent Gateway, system architects again have the option of deploying GeoEvent Server with multiple machines configured as part of a single ArcGIS Server site. The recommendation is still that your ArcGIS Enterprise be deployed separately allowing you the flexibility to scale the GIS Server component for map and feature services.

The only real change to the diagram would be to de-emphasize the white space between GeoEvent Server instances:

Above, I'm illustrating that a GeoEvent Gateway, running on each ArcGIS Server machine, will coordinate through the ArcGIS Server site dedicated for real-time event processing to distribute events processing across the three machines.

There are still a number of questions to prove out here. What if a solution were being designed which had several, independent, inbound feeds whose velocity and/or event volume were each nominal (no more than 1,000 or 2,000 events per second)? Such a use case wouldn't require an event distribution mechanism to leverage several machines. It might be better, in this case, to follow the 10.5 / 10.5.1 pattern of deploying independent instances of GeoEvent Server in separate sites, dedicating instances to handle one or two independent inbound data streams. The above single-site, multi-machine approach would be recommended only if you were deploying 10.6 and had a single inbound event stream whose velocity / event volume were too much for a single server to handle.

A tutorial is being prepared for system architects and solution engineers which will identify recommended best practices for product deployment in a multi-machine configuration. The tutorial should be available by the Developer Summit, March 6 - 9 2018. Please reach out to Greg Tieman‌ with questions on this. You can read more about the GeoEvent Gateway in the on-line help which was pushed out 9-January: What's New at 10.6 GeoEvent Server

(Internationalization of the on-line help is still pending as I post this reply...)

- RJ

Re: GeoEvent and High Availibility

JohnDye2 — Tue, 06 Feb 2018 21:57:03 GMT

Sorry, I'm a little thick sometimes.

Just for my sanity:

If your solution has multiple inbound feeds with an event velocity and/or volume that did not exceed 2,000 events per second, it is recommended to deploy multiple single-machine geoevent sites and dedicate each to a one or two inbound feeds.
If your solution includes inbound feeds where the event velocity of volume does exceed 2,000 events per second, then an event distribution mechanism is likely needed and as such, a multi-machine geoevent site is recommended so that the geoevent-gateway can distribute events among the machines.

Is that correct?

Re: GeoEvent and High Availibility

RJSunderman — Wed, 07 Feb 2018 01:45:05 GMT

John Dye‌ ... you have correctly interpreted the recommendation.

You probably want to defer production engagements you think require GeoEvent multi-machine functionality and continue to deploy GeoEvent Server in separate ArcGIS Server sites. There's still work we need to complete to document best practices for you. Also, there are some issues we are investigating which tend to manifest with deliberate fault injection (like abruptly powering down a machine or severing its network connection). There will likely be a patch for 10.6 for multi-machine deployments ... stay tuned.

- RJ

Re: GeoEvent and High Availibility

JohnDye2 — Wed, 07 Feb 2018 16:21:32 GMT

Thanks RJ. I suppose the one question your responses then raises is whether or not there will be a streamlined process for moving from a GeoEvent single-machine-multi-site configuration to a multi-machine-single-site configuration once this is all said and done.

It sure would be a headache to go with the former only to have to undertake major efforts to move to the latter at say, 10.6.1 or 10.7

Re: GeoEvent and High Availibility

RJSunderman — Wed, 07 Feb 2018 22:22:03 GMT

I normally prefer exporting my GeoEvent configuration to an XML file, then uninstalling GeoEvent Server and installing the newer release (vs. following an upgrade-in-place workflow). Product upgrades make more sense for ArcGIS Server and Portal for ArcGIS where exporting and importing your site configuration can be more difficult. For GeoEvent Server, it's really easy to snapshot the configuration as XML and then selectively import just the inputs, outputs, GeoEvent Services (etc.) that you know you're using - or import the entire configuration.

In the case you're describing, where you want to scale your deployment from a single-machine architecture to a multi-machine architecture ... I would definitely recommend you export the GeoEvent configuration, perform whatever s/w upgrades are needed to move to 10.6, establish your ArcGIS for Server site with its multiple machines and multiple instances of GeoEvent Server, then import your GeoEvent configuration once, from the XML. I would then test to make sure that all GeoEvent Server instances 1) receive the configured elements; and 2) event processing is distributed across the machines in the site when event data is sent to any one machine's input.

- RJ

Re: GeoEvent and High Availibility

DavidMartin — Wed, 11 Apr 2018 08:38:35 GMT

Thanks for the useful pointers. The conversation (at 10.6) focuses on SCALABILITY to cope with:

high message volumes resulting from accumulation from multiple sources
- where the recommendation appears to be to use multiple single-machine sites
  - so that different sites can consume messages from different sources
high message volumes from large single sources
- where the recommendation appears to be to use a multiple-machine site (perhaps waiting first for a patch!)
  - so that the messages can be load-balanced across machines

However, what about solution AVAILABILITY, which could manifest as a requirement even when consuming very low message volumes, but where the business criticality of the solution is high?

I assume that this is best met using a multiple-machine site (once patched)? There appears to be a recommendation, however, that multiple-machine sites should have a minimum of 3 GeoEvent Servers. This could feel a bit excessive where message volumes are low. QUESTION 1: Is there a good reason why a multiple-machine site can't just have 2 GeoEvent Servers?

Another caveat with a multiple-machine site appears to be that outputs cannot include Stream Services. As such, there will be a need to store geoevents prior to displaying them in a client, and the client will need to poll a feature service to retrieve them. QUESTION 2: Is that assumption correct?

That then gives rise to the question of whether the spatiotemporal data store (with a potentially higher-cost, multi-node deployment) is overkill for low volume messages. QUESTION 3: Do you have any recommendations as to when to consider deployment of the spatiotemporal data store? (e.g. a threshold level of throughput).

That's probably enough questions for one post... 🙂

Re: GeoEvent and High Availibility

RJSunderman — Wed, 11 Apr 2018 19:26:24 GMT

Hello David –

>> However, what about solution AVAILABILITY, which could manifest as a requirement even when consuming very low message volumes, but where the business criticality of the solution is high? I assume that this is best met using a multiple-machine site (once patched)?

Perhaps … there are multiple considerations which is why I recommend folks who are looking at scaling out real-time solutions work with their Esri Technical Advisor or contract with Esri Professional Services for consultation. I’ll try to answer you briefly here however.

Depending on the type of input you are using, a multiple-machine architecture might provide the failover you are looking for. Our testing when using an input which polls an external source for data was very positive. If one machine in a multi-machine site were to go off-line we’ve confirmed that another machine will “adopt” and being running the input so that event ingest doesn’t terminate with the one machine.

If you’re using an input like a JSON REST receiver, or the TCP/Text input, to push data to GeoEvent, then the burden is on the data provider to specify which machine, endpoint, port, (etc.) to push the data to. The challenge here is that there is no built-in way to signal an external data provider that a particular GeoEvent Server instance has gone off-line, so it is difficult to know when to direct data to a different machine or endpoint. In this case a multi-machine setup is probably not going to help; a significant amount of custom development would be required on your part as a solution architect.

I might recommend, for the case where you have low event velocity / volume but message data is critical, that you set up multiple independent GeoEvent Server instances to provide basic redundancy. Process the critical inbound data in parallel across multiple redundant instances and write the resulting feature records out to parallel feature services. The challenge, then, is advising consuming clients when to use the primary feature layer and when to switch to a failover.

Please keep in mind that GeoEvent Server, at a fundamental level, was architected with the assumption that data from sensor networks would be both frequent and periodic. If a reported temperature reading is missed or dropped it’s not a big deal because another will come along from the sensor momentarily.

This assumption is not entirely compatible with stipulations that every event record is critical, neither frequent nor periodic, and data loss cannot be tolerated. If GeoEvent Server receives the data the system will reliability process and disseminate event and feature records. Network stability, datababase failure, feature service availability and recycling … there are many different points at which a system which distributes capability across multiple components might result in a loss of data. System architecture and solution design can address these, but I do not want to take this discussion into one of general “high availability” which often means different things to different customers in different contexts.

>> There appears to be a recommendation, however, that multiple-machine sites should have a minimum of 3 GeoEvent Servers. This could feel a bit excessive where message volumes are low. Is there a good reason why a multiple-machine site can't just have 2 GeoEvent Servers?

Yes, and it has more to do with general computer science principals than anything specific to GeoEvent Server. If two nodes are receiving input and one goes off-line, when it comes back there is no majority or quorum to stipulate that the node coming back on-line is out-of-date. You are left with two peers, each thinking they have the latest state. When you have three (or more) nodes and one fails, upon recovery the other two can establish for the node coming back on-line what “latest” looks like. So, the recommendation for three GeoEvent Server instances is to avoid a “split-brain” condition.

>> Another caveat with a multiple-machine site appears to be that outputs cannot include Stream Services. As such, there will be a need to store geoevents prior to displaying them in a client, and the client will need to poll a feature service to retrieve them. Is that assumption correct?

No, you should be able to leverage stream services in either a single-machine or a multi-machine deployment.

We did find a bug in 10.6 where consumers subscribing to web socket endpoints exposed by machines in a multiple machine deployment would sometimes not receive any feature records. The reason was that the web socket they subscribed to was not actually broadcasting any data. If the client were a web map, for example, a few manual “F5” refresh requests would eventually have the client subscribe via the server instance that actually published the stream service, which was the only one whose web socket was actually broadcasting feature records.

We’ve addressed that issue and the “primary” stream service is now using the message bus to “fan out” and send feature records to the other stream service instances so that a client can subscribe to any machine’s web socket and get all of the feature records processed across the multiple machines – regardless of which machine received an event record and which machine actually processed the event record. This fix will be included in our first patch for the 10.6 release.

>> Do you have any recommendations as to when to consider deployment of the spatiotemporal data store? (e.g. a threshold level of throughput).

To quote an associate, you know you’re dealing with big data when your traditional methods and tools begin to fail. Many folks successfully leverage feature services backed by a traditional relational database (RDBMS) and use the Portal’s hosting server’s managed database to persist their feature records.

Where reliance on an RDBMS begins to fail is when you need to persist more than about 200 events each second. There is a lot of overhead to the REST requests GeoEvent Server has to make on a feature service’s add (or update) feature endpoint to initiate a transaction with a relational data store. Depending on network and database tuning you may discover that transactional latency allows you to persist fewer event records each second. This is the primary reason to consider switching and being using the spatiotemporal big data store.

GeoEvent Server discovers, through Portal and the Portal hosting server, credentials needed to open a socket connection directly to Elasticsearch (the no-SQL database behind the spatiotemporal big data store). Avoiding REST requests means that feature records can be persisted at much higher velocity into the database.

A second reason to consider using the spatiotemporal big data store is that Elasticsearch natively provides data replication across multiple nodes when you establish a data ring by installing ArcGIS Data Store (and selecting the spatiotemporal big data store capability) on multiple machines.

If you include three nodes of the spatiotemporal big data store in your architecture – again, trying to avoid the potential of a split-brain condition – and one of your database server machines were to fail, only one node of the data ring would go off-line. Client requests for data will be redirected to surviving instances. Replicas of your feature records, created automatically and managed by Elasticsearch, will be returned to requesting clients.

Hope this information helps –

Josh Joyner‌

Re: GeoEvent and High Availibility

DavidMartin — Thu, 12 Apr 2018 08:21:08 GMT

Thank you Josh - I couldn't have asked for a more comprehensive reply!

Just 2 further questions arise from your response:

1/ Is there a scheduled release date (or approximate ETA) for the 10.6 patch?

2/ In a multiple-machine site (minimum 3 nodes), should we expect to have to license each and every node with a GeoEvent Server license? (i.e. pay for 3 licenses)

Re: GeoEvent and High Availibility

DavidMartin — Thu, 12 Apr 2018 09:02:37 GMT

Actually, I have another...:-)

If you’re using an input like a JSON REST receiver, or the TCP/Text input, to push data to GeoEvent, then the burden is on the data provider to specify which machine, endpoint, port, (etc.) to push the data to.

I take it from this that a multiple-machine site exposes no single endpoint for receipt of an http request (whether POST or GET)? If that's the case, then I guess a suitable load balancer needs to be added up-front which can health-check each machine in order to distribute such requests only to healthy ones? Which could just as easily be done with a suite of single-machine sites, for inputs of this nature, provided no gap detection, geo-fencing or similar is required?

For "guaranteed" receipt, the onus would then just remain on the sender (or middleware) to re-send if no 200 OK response is received from GeoEvent Server?

Re: GeoEvent and High Availibility

RJSunderman — Thu, 12 Apr 2018 22:09:58 GMT

>> Is there a scheduled release date (or approximate ETA) for the 10.6 patch?

We are working now to determine what will be committed for a 10.6.1 final release, what portion of that will be ported back for 10.6 Patch 1 … and if anything critical must be released as part of the patch which cannot be committed to 10.6.1 (not a configuration we desire, but sometimes have to do).

I would look for 10.6 Patch 1 to be released by UC in July, but it may come out earlier, as soon as 10.6.1 is handed off to the Esri Release Team for final certification and testing.

>> In a multiple-machine site (minimum 3 nodes), should we expect to have to license each and every node with a GeoEvent Server license? (i.e. pay for 3 licenses)

Yes, every deployment / installation of GeoEvent Server must be licensed using (at least) the GeoEvent Server license for that server role. Optionally you can also apply an ArcGIS Server standard / advanced license to the machine you’ve deployed GeoEvent Server, if you want enable the full capability of the ArcGIS Server on that machine – but it is sufficient to license both the ArcGIS Server and GeoEvent Server using a GeoEvent Server license role if you only want the real-time capabilities on that machine.

The same, by the way, is not true for ArcGIS Data Store. With a single ArcGIS Enterprise license you can install ArcGIS Data Store on as many machines as you like, configuring each as a node of the spatiotemporal big data store registered with the Portal’s hosting GIS Server, without having to purchase additional licenses for the instances of ArcGIS Data Store you are deploying.

>> I take it from this that a multiple-machine site exposes no single endpoint for receipt of an http request (whether POST or GET)? If that's the case, then I guess a suitable load balancer needs to be added up-front which can health-check each machine in order to distribute such requests only to healthy ones?

That is correct. Nothing related to the GeoEvent Gateway, GeoEvent Server, or ArcGIS Server components provide a single endpoint (in a multiple-machine architecture) to which a data provider can direct data when actively pushing data to GeoEvent Server. Yes, you can probably configure a load balancer with a “sticky session” to establish affinity with a particular instance of GeoEvent Server, using one of the monitoring endpoints in the GeoEvent Server Admin API to determine if that instance is still available. But I’m not the one to advise you on how to do that – I’ve no experience in that area of system architecture.

>> Which could just as easily be done with a suite of single-machine sites, for inputs of this nature, provided no gap detection, geo-fencing or similar is required?

Yes, as I understand load balancing technology, you could use a LB to route event records to a single-machine “silo’d” instance of GeoEvent Server just as easily as you could an instance participating in a multi-machine configuration. The drawback would be that the single machine instance would not have any peers to which it could further distribute event records for real-time event processing. Also, as you note, if the LB did not guarantee that all tracked assets (those with a given TRACK_ID) were routed to the same machine, real-time analytics like Track Gap Detection, Incident Detection, and spatial operations such as ENTER and EXIT will not work consistently as one machine's "state" (i.e. knowledge of a prior event record) is not propagated or shared between machines in a multi-machine site. Geofencing would work, as long as you had explicitly loaded a consistent set of geofences to each GeoEvent Server instance (in a single-machine "silo'd" configuration).

>> For "guaranteed" receipt, the onus would then just remain on the sender (or middleware) to re-send if no 200 OK response is received from GeoEvent Server?

No, unfortunately that won’t work. An HTTP / 200 response does not imply success in constructing or handling of an event record.

Every inbound connector has two components: an adapter and a transport. It is the transport’s responsibility to receive a payload (think “raw bytes”) and confirm with the data provider that the data was successfully received. That’s all the HTTP / 200 response means.

It is the adapter’s job to reference a schema (e.g. GeoEvent Definition) and construct an event record from the payload provided by its associated transport. That’s why you can configure new connectors to, say, receive JSON over TCP – when out-of-the-box there is only a receive delimited Text over TCP – without writing any code. You are just configuring a new inbound connector paring an out-of-the-box transport with an adapter.

The HTTP / 200 returned from an inbound connector, by its transport, does not mean that the inbound connector’s adapter was successfully able to instantiate an event record for processing, that the event record was successfully processed by a GeoEvent Service (configuration errors in a processor might end up throwing exceptions) or that an output was able to disseminate the event record’s data.

- RJ

Re: GeoEvent and High Availibility

DavidMartin — Fri, 13 Apr 2018 11:26:28 GMT

Again RJ, thanks for a very comprehensive reply. (And sorry for calling you Josh before!)

Just one point has confused me a little. In suggesting use of a Load Balancer (LB) to act as a single endpoint for inputs that are actively sent to GeoEvent Server (such as http requests - with input data packaged in a POST / GET), I was imagining that the LB would distribute those requests across machines in a multiple-machine site. As all machines in such a site share the same configuration, any one of them can presumably handle the input?

Your suggestion that a "sticky session" might be required leads me to believe there might be a problem with my original understanding, and makes me wonder what the point would be of having the LB in the first place?

I do appreciate, however, that where stateful processing is required (Track Gap Detection, Incident Detection, and spatial operations such as ENTER and EXIT), use of an LB in this fashion would not be suitable, as multiple-machine sites don't share a track's last-known state across machines.

My summary understanding:

A multiple-machine GeoEvent Server site offers:

- shared configuration (input/output connectors, processors etc) - provided by Zookeeper

- event distribution - provided by Kafka

... but does NOT offer:

- shared track state (a site-level understanding of a track's last-known state)

- a single endpoint to which inputs can actively be sent

Re: GeoEvent and High Availibility

RJSunderman — Fri, 13 Apr 2018 21:19:32 GMT

David Martin‌ - I think your summary / take-away is accurate. A load balancer could be used to simply round-robin event records to available machines because, yes, the inputs on all the machines are configured the same. Doing this may not be optimal as the Kafka event distribution at 10.6 will still attempt to do *its* job and distribute event records with a given TRACK_ID to a single server ... with fail-over to another server only if the primary goes off-line.

To clarify, as long as all machines remain up and running, the fact that "state" is not shared between machines does not matter. Kafka is handling event distribution according to TRACK_ID and is sending event records with a given TRACK_ID to a consistent server. It is only when a machine fails and Kafka has to choose a another machine to process event records with that TRACK_ID that event records previously handled by the machine that failed may produce false positive analytics ... since the secondary machine (which has never seen event records with *these* TRACK_ID values before) has to begin to accumulate a history of the event records it is now receiving.

The point I was trying to make is that -- if you elect to handle event distribution yourself, using a round-robin approach across multiple independent "silo'd" instances -- Track Gap Detection, Incident Detection, and spatial operations such as ENTER and EXIT are not going to work because event records with a given TRACK_ID are not being routed to a consistent machine.

Also, the real purpose of the load balancer is not traditional load balancing. The purpose in this case would be to expose a single receiving endpoint (the URL of the LB) to data providers and obscure (from data providers) the fact that multiple endpoints are available. This potentially provides your solution some resiliency when a given machine fails ... ingest does not come to a screeching halt when data providers continue trying to send data to the failed machine's endpoint(s). Your LB, through a "sticky session", would choose which machine should act as the primary ingest node for a given inbound data stream and fail-over to a secondary machine only if the first machine became unavailable.

- RJ

Re: GeoEvent and High Availibility

DavidMartin — Sat, 14 Apr 2018 06:37:22 GMT

Now I get it. I hadn't fully appreciated the extent of the event distribution role that Kafka was performing in allocating events with a common TRACK_ID to an individual server, with failover for resilience. It's presumably a reason to be careful about how (/if?) you populate TRACK_ID on your events, as unless you actually need to maintain track state, it could otherwise reduce the effectiveness of event distribution in a multiple-machine site?

Thank you, rsunderman-esristaff‌. I couldn't have asked for better support.

Re: GeoEvent and High Availibility

NickBoon3 — Mon, 15 Apr 2019 12:03:08 GMT

Hi RJ Sunderman‌

I am interested in continuing the discussion that David Martin started and wondered if there have been any more developments with 10.7 with regards to using a multi-machine single site deployment of GeoEvent Server for stream services?

We would like to deploy a multi-GeoEvent Server machine site to subscribe to a number of JMS endpoints, process the feeds and provide 1+ Stream and feature services to Web clients. Are there any deployment patterns that Esri recommend as we are getting mixed messages from the documentation regarding stream services.

10.6.1 High Availability Tutorial (Page 23): Stream services only run on a single machine.

Thanks

Nick

Re: GeoEvent and High Availibility

RJSunderman — Mon, 15 Apr 2019 18:05:03 GMT

Hello Nick -

The reference you highlight, that stream services only run on a single machine, is intended to convey that the JVM used to run GeoEvent Server is the container used to run stream services (unlike traditional map/feature services which have multiple instances and are run by SOC components managed by ArcGIS Server). So the server machine used to publish a stream service is the machine whose GeoEvent Server JVM is running the stream service. However, as I mention above, copies of the Esri Feature JSON records a stream service broadcasts are forwarded to other GeoEvent Server instances in an ArcGIS Server site. This “fan out” allows web clients to subscribe to any machine’s stream service and get all of the feature records processed across a single site, multiple machine solution – regardless of which machine received an event record and which machine actually processed the event record.

A bug (BUG-000114373) with the "fan out" mechanism was addressed with the 10.6.1 release and ported back for release as part of ArcGIS GeoEvent Server 10.6 Patch 2.

The 10.7 release does not include any specific changes that, I think, should influence an system architect to choose a "silo" approach (where multiple independent instances of GeoEvent Server are run, each in their own ArcGIS Server site) vs. a "site" approach whose architecture deploys multiple GeoEvent Server instances configured as part of a single ArcGIS Server site.

If scalability is your primary objective, I would probably recommend the "silo" approach. The solution you architect would need to include an external Apache Kafka (or similar message broker) to handle event record distribution across the multiple independent ArcGIS Server / GeoEvent Server instances. Adopting and maintaining your own Kafka solution introduces its own technical burden, but I think we are finding that, for scalability, this approach lends itself to a system that is easier to maintain and administer overall.

If reliability and fault-tolerance is your primary objective (I really dislike using the term high-availability) then a "site" approach is an option. The solution you architect could deploy multiple ArcGIS Server / GeoEvent Server instances in a single site and rely on the Apache Kafka and Zookeeper built-in to the GeoEvent Gateway to mitigate individual machine failures. But a reliability objective can also be addressed with an an active / active approach using multiple independent instances of GeoEvent Server to build redundancy into a distributed solution for fault-tolerance.

There are advantages to going with the single site, multiple machines approach for resiliency. Specifically when your solution relies on GeoEvent Server polling external web servers / web services for data (which you mentioned), we have observed that the GeoEvent Gateway reliably mitigates the failure of a single node -- which had been the node polling for input -- by allowing another node to "adopt" the input and begin handling the data polling activity.

There are also significant system complexity and administration disadvantages to the "site" approach, which is why I recommend folks who are considering multiple machine, distributed architectures for real-time solutions work with their Esri Technical Advisor or contract with Esri Professional Services for consultation. Only after discussing specific objectives and weighing both the pros and cons can a recommendation be made as to which approach your solution ought to take.

- RJ

Re: GeoEvent and High Availibility

NickBoon3 — Tue, 16 Apr 2019 07:42:19 GMT

Hi RJ, Thank you for a very quick and comprehensive reply which has clarified what I thought/hoped! We are interested in a reliable/fault-tolerant system so the multi-machine GE Site is our preferred pattern. We have engaged with our local Esri professional services team and will be used to validate any design decision I or my team make.

If I am lucky enough to get to the UC this year I will buy you a beer.

Kind regards

Nick