nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

GeoEvent and High Availibility

AzinSharaf

This would be our Portal and ArcGIS Server high availability deployment.

We want to add a HA GeoEvent Server to this architecture. What would be the best solution based on the following limitations/recommendations?

1- Don't use multi-machine sites with GeoEvent

2- Don't install GeoEvent in your base GIS site

If we create two independent GeoEvent site, we should sync them manually? Any better solution?

Trouver plus de publications taguées avec

Accepted answers

All comments

RJSunderman

Hello Azin -

The ArcGIS Server documentation includes reference architectures for deploying additional server roles (such as GeoEvent Server) to your ArcGIS Enterprise. The terms I'm using here are consistent with the 10.5.x product release. There are reference architectures for high availability as well as for GeoEvent Server.

Based on the icons you included in your system architecture illustration, I might assume you've seen these references - but I wanted to make sure.

I would point out that the reference architecture for GeoEvent Server which depicts a horizontal scale-out deliberately has white space between the machines:

Additional server deployment - GeoEvent Server

The white space is intended to depict your first stipulation -- that you do not 'Join Existing Site' when installing the GIS Server component of ArcGIS Server with the intention of later installing GeoEvent Server. Each of the three GeoEvent Server machines depicted above exist in their own site. They do not know anything about one another. This is to prevent leveraging a concept we refer to as "GeoEvent Clustering" which we introduced with the 10.3 release. Changes to address some issues were included with the 10.4 release. We eventually realized that the concept was fundamentally not working as we intended, and the advice now for releases 10.3 / 10.4 / 10.5 is to not have more than one GIS Server in a site when GeoEvent Server is installed.

We are working on developing a "Gateway" component for 10.6 with the intent that GeoEvent Server will again offer some resiliency and improved scalability. However, until then, your challenge is to figure out how to configure an external component to handle event distribution to two or more instances of GeoEvent Server which do not know anything about one another.

We prepared a GeoEvent Server Resiliency tutorial which proposes using Kafka as an external message broker. Kafka provides load balancing to your real-time event message ingest by holding messages in a queue until consumers (individual GeoEvent Server instances) fetch them at the rate they are able to process them. My terminology may be off; I would recommend you refer to Kafka's documentation as you read through the GeoEvent Server Resiliency tutorial.

I hope this information is helpful --

AzinSharaf

Thanks rsunderman-esristaff

HarroldSompotan

Hi RJ,

Now that Arcgis for Server and Geoevent Server 10.6 is available, and the Gateway components are in place, are we still encouraging other to go with a siloed Arcgis Geoevent Server instead of Arcgis Geoevent Server in a multimachine site?

Currently the documentation at 10.6 still mentions below:

"It is NOT recommended to use multi-machine sites with ArcGIS GeoEvent Server. Instead, you should configure multiple single-machine sites to dedicate ArcGIS GeoEvent Server machine resources to specific real-time data streams:"

Deployment patterns for ArcGIS Enterprise—ArcGIS Enterprise | ArcGIS Enterprise

RJSunderman

Yes, with the introduction of the GeoEvent Gateway, system architects again have the option of deploying GeoEvent Server with multiple machines configured as part of a single ArcGIS Server site. The recommendation is still that your ArcGIS Enterprise be deployed separately allowing you the flexibility to scale the GIS Server component for map and feature services.

The only real change to the diagram would be to de-emphasize the white space between GeoEvent Server instances:

Above, I'm illustrating that a GeoEvent Gateway, running on each ArcGIS Server machine, will coordinate through the ArcGIS Server site dedicated for real-time event processing to distribute events processing across the three machines.

There are still a number of questions to prove out here. What if a solution were being designed which had several, independent, inbound feeds whose velocity and/or event volume were each nominal (no more than 1,000 or 2,000 events per second)? Such a use case wouldn't require an event distribution mechanism to leverage several machines. It might be better, in this case, to follow the 10.5 / 10.5.1 pattern of deploying independent instances of GeoEvent Server in separate sites, dedicating instances to handle one or two independent inbound data streams. The above single-site, multi-machine approach would be recommended only if you were deploying 10.6 and had a single inbound event stream whose velocity / event volume were too much for a single server to handle.

A tutorial is being prepared for system architects and solution engineers which will identify recommended best practices for product deployment in a multi-machine configuration. The tutorial should be available by the Developer Summit, March 6 - 9 2018. Please reach out to Greg Tieman‌ with questions on this. You can read more about the GeoEvent Gateway in the on-line help which was pushed out 9-January: What's New at 10.6 GeoEvent Server

(Internationalization of the on-line help is still pending as I post this reply...)

- RJ

johnd-usaa

Sorry, I'm a little thick sometimes.

Just for my sanity:

If your solution has multiple inbound feeds with an event velocity and/or volume that did not exceed 2,000 events per second, it is recommended to deploy multiple single-machine geoevent sites and dedicate each to a one or two inbound feeds.
If your solution includes inbound feeds where the event velocity of volume does exceed 2,000 events per second, then an event distribution mechanism is likely needed and as such, a multi-machine geoevent site is recommended so that the geoevent-gateway can distribute events among the machines.

Is that correct?

DavidMartin

Thanks for the useful pointers. The conversation (at 10.6) focuses on SCALABILITY to cope with:

high message volumes resulting from accumulation from multiple sources
- where the recommendation appears to be to use multiple single-machine sites
  - so that different sites can consume messages from different sources
high message volumes from large single sources
- where the recommendation appears to be to use a multiple-machine site (perhaps waiting first for a patch!)
  - so that the messages can be load-balanced across machines

However, what about solution AVAILABILITY, which could manifest as a requirement even when consuming very low message volumes, but where the business criticality of the solution is high?

I assume that this is best met using a multiple-machine site (once patched)? There appears to be a recommendation, however, that multiple-machine sites should have a minimum of 3 GeoEvent Servers. This could feel a bit excessive where message volumes are low. QUESTION 1: Is there a good reason why a multiple-machine site can't just have 2 GeoEvent Servers?

Another caveat with a multiple-machine site appears to be that outputs cannot include Stream Services. As such, there will be a need to store geoevents prior to displaying them in a client, and the client will need to poll a feature service to retrieve them. QUESTION 2: Is that assumption correct?

That then gives rise to the question of whether the spatiotemporal data store (with a potentially higher-cost, multi-node deployment) is overkill for low volume messages. QUESTION 3: Do you have any recommendations as to when to consider deployment of the spatiotemporal data store? (e.g. a threshold level of throughput).

That's probably enough questions for one post... 🙂

RJSunderman

Hello David –

>> However, what about solution AVAILABILITY, which could manifest as a requirement even when consuming very low message volumes, but where the business criticality of the solution is high? I assume that this is best met using a multiple-machine site (once patched)?

Perhaps … there are multiple considerations which is why I recommend folks who are looking at scaling out real-time solutions work with their Esri Technical Advisor or contract with Esri Professional Services for consultation. I’ll try to answer you briefly here however.

Depending on the type of input you are using, a multiple-machine architecture might provide the failover you are looking for. Our testing when using an input which polls an external source for data was very positive. If one machine in a multi-machine site were to go off-line we’ve confirmed that another machine will “adopt” and being running the input so that event ingest doesn’t terminate with the one machine.

If you’re using an input like a JSON REST receiver, or the TCP/Text input, to push data to GeoEvent, then the burden is on the data provider to specify which machine, endpoint, port, (etc.) to push the data to. The challenge here is that there is no built-in way to signal an external data provider that a particular GeoEvent Server instance has gone off-line, so it is difficult to know when to direct data to a different machine or endpoint. In this case a multi-machine setup is probably not going to help; a significant amount of custom development would be required on your part as a solution architect.

I might recommend, for the case where you have low event velocity / volume but message data is critical, that you set up multiple independent GeoEvent Server instances to provide basic redundancy. Process the critical inbound data in parallel across multiple redundant instances and write the resulting feature records out to parallel feature services. The challenge, then, is advising consuming clients when to use the primary feature layer and when to switch to a failover.

Please keep in mind that GeoEvent Server, at a fundamental level, was architected with the assumption that data from sensor networks would be both frequent and periodic. If a reported temperature reading is missed or dropped it’s not a big deal because another will come along from the sensor momentarily.

This assumption is not entirely compatible with stipulations that every event record is critical, neither frequent nor periodic, and data loss cannot be tolerated. If GeoEvent Server receives the data the system will reliability process and disseminate event and feature records. Network stability, datababase failure, feature service availability and recycling … there are many different points at which a system which distributes capability across multiple components might result in a loss of data. System architecture and solution design can address these, but I do not want to take this discussion into one of general “high availability” which often means different things to different customers in different contexts.

>> There appears to be a recommendation, however, that multiple-machine sites should have a minimum of 3 GeoEvent Servers. This could feel a bit excessive where message volumes are low. Is there a good reason why a multiple-machine site can't just have 2 GeoEvent Servers?

Yes, and it has more to do with general computer science principals than anything specific to GeoEvent Server. If two nodes are receiving input and one goes off-line, when it comes back there is no majority or quorum to stipulate that the node coming back on-line is out-of-date. You are left with two peers, each thinking they have the latest state. When you have three (or more) nodes and one fails, upon recovery the other two can establish for the node coming back on-line what “latest” looks like. So, the recommendation for three GeoEvent Server instances is to avoid a “split-brain” condition.

>> Another caveat with a multiple-machine site appears to be that outputs cannot include Stream Services. As such, there will be a need to store geoevents prior to displaying them in a client, and the client will need to poll a feature service to retrieve them. Is that assumption correct?

No, you should be able to leverage stream services in either a single-machine or a multi-machine deployment.

We did find a bug in 10.6 where consumers subscribing to web socket endpoints exposed by machines in a multiple machine deployment would sometimes not receive any feature records. The reason was that the web socket they subscribed to was not actually broadcasting any data. If the client were a web map, for example, a few manual “F5” refresh requests would eventually have the client subscribe via the server instance that actually published the stream service, which was the only one whose web socket was actually broadcasting feature records.

We’ve addressed that issue and the “primary” stream service is now using the message bus to “fan out” and send feature records to the other stream service instances so that a client can subscribe to any machine’s web socket and get all of the feature records processed across the multiple machines – regardless of which machine received an event record and which machine actually processed the event record. This fix will be included in our first patch for the 10.6 release.

>> Do you have any recommendations as to when to consider deployment of the spatiotemporal data store? (e.g. a threshold level of throughput).

To quote an associate, you know you’re dealing with big data when your traditional methods and tools begin to fail. Many folks successfully leverage feature services backed by a traditional relational database (RDBMS) and use the Portal’s hosting server’s managed database to persist their feature records.

Where reliance on an RDBMS begins to fail is when you need to persist more than about 200 events each second. There is a lot of overhead to the REST requests GeoEvent Server has to make on a feature service’s add (or update) feature endpoint to initiate a transaction with a relational data store. Depending on network and database tuning you may discover that transactional latency allows you to persist fewer event records each second. This is the primary reason to consider switching and being using the spatiotemporal big data store.

GeoEvent Server discovers, through Portal and the Portal hosting server, credentials needed to open a socket connection directly to Elasticsearch (the no-SQL database behind the spatiotemporal big data store). Avoiding REST requests means that feature records can be persisted at much higher velocity into the database.

A second reason to consider using the spatiotemporal big data store is that Elasticsearch natively provides data replication across multiple nodes when you establish a data ring by installing ArcGIS Data Store (and selecting the spatiotemporal big data store capability) on multiple machines.

If you include three nodes of the spatiotemporal big data store in your architecture – again, trying to avoid the potential of a split-brain condition – and one of your database server machines were to fail, only one node of the data ring would go off-line. Client requests for data will be redirected to surviving instances. Replicas of your feature records, created automatically and managed by Elasticsearch, will be returned to requesting clients.

Hope this information helps –

Josh Joyner‌

DavidMartin

Thank you Josh - I couldn't have asked for a more comprehensive reply!

Just 2 further questions arise from your response:

1/ Is there a scheduled release date (or approximate ETA) for the 10.6 patch?

2/ In a multiple-machine site (minimum 3 nodes), should we expect to have to license each and every node with a GeoEvent Server license? (i.e. pay for 3 licenses)

DavidMartin

Actually, I have another...:-)

If you’re using an input like a JSON REST receiver, or the TCP/Text input, to push data to GeoEvent, then the burden is on the data provider to specify which machine, endpoint, port, (etc.) to push the data to.

I take it from this that a multiple-machine site exposes no single endpoint for receipt of an http request (whether POST or GET)? If that's the case, then I guess a suitable load balancer needs to be added up-front which can health-check each machine in order to distribute such requests only to healthy ones? Which could just as easily be done with a suite of single-machine sites, for inputs of this nature, provided no gap detection, geo-fencing or similar is required?

For "guaranteed" receipt, the onus would then just remain on the sender (or middleware) to re-send if no 200 OK response is received from GeoEvent Server?

RJSunderman

>> Is there a scheduled release date (or approximate ETA) for the 10.6 patch?

We are working now to determine what will be committed for a 10.6.1 final release, what portion of that will be ported back for 10.6 Patch 1 … and if anything critical must be released as part of the patch which cannot be committed to 10.6.1 (not a configuration we desire, but sometimes have to do).

I would look for 10.6 Patch 1 to be released by UC in July, but it may come out earlier, as soon as 10.6.1 is handed off to the Esri Release Team for final certification and testing.

>> In a multiple-machine site (minimum 3 nodes), should we expect to have to license each and every node with a GeoEvent Server license? (i.e. pay for 3 licenses)

Yes, every deployment / installation of GeoEvent Server must be licensed using (at least) the GeoEvent Server license for that server role. Optionally you can also apply an ArcGIS Server standard / advanced license to the machine you’ve deployed GeoEvent Server, if you want enable the full capability of the ArcGIS Server on that machine – but it is sufficient to license both the ArcGIS Server and GeoEvent Server using a GeoEvent Server license role if you only want the real-time capabilities on that machine.

The same, by the way, is not true for ArcGIS Data Store. With a single ArcGIS Enterprise license you can install ArcGIS Data Store on as many machines as you like, configuring each as a node of the spatiotemporal big data store registered with the Portal’s hosting GIS Server, without having to purchase additional licenses for the instances of ArcGIS Data Store you are deploying.

>> I take it from this that a multiple-machine site exposes no single endpoint for receipt of an http request (whether POST or GET)? If that's the case, then I guess a suitable load balancer needs to be added up-front which can health-check each machine in order to distribute such requests only to healthy ones?

That is correct. Nothing related to the GeoEvent Gateway, GeoEvent Server, or ArcGIS Server components provide a single endpoint (in a multiple-machine architecture) to which a data provider can direct data when actively pushing data to GeoEvent Server. Yes, you can probably configure a load balancer with a “sticky session” to establish affinity with a particular instance of GeoEvent Server, using one of the monitoring endpoints in the GeoEvent Server Admin API to determine if that instance is still available. But I’m not the one to advise you on how to do that – I’ve no experience in that area of system architecture.

>> Which could just as easily be done with a suite of single-machine sites, for inputs of this nature, provided no gap detection, geo-fencing or similar is required?

Yes, as I understand load balancing technology, you could use a LB to route event records to a single-machine “silo’d” instance of GeoEvent Server just as easily as you could an instance participating in a multi-machine configuration. The drawback would be that the single machine instance would not have any peers to which it could further distribute event records for real-time event processing. Also, as you note, if the LB did not guarantee that all tracked assets (those with a given TRACK_ID) were routed to the same machine, real-time analytics like Track Gap Detection, Incident Detection, and spatial operations such as ENTER and EXIT will not work consistently as one machine's "state" (i.e. knowledge of a prior event record) is not propagated or shared between machines in a multi-machine site. Geofencing would work, as long as you had explicitly loaded a consistent set of geofences to each GeoEvent Server instance (in a single-machine "silo'd" configuration).

>> For "guaranteed" receipt, the onus would then just remain on the sender (or middleware) to re-send if no 200 OK response is received from GeoEvent Server?

No, unfortunately that won’t work. An HTTP / 200 response does not imply success in constructing or handling of an event record.

Every inbound connector has two components: an adapter and a transport. It is the transport’s responsibility to receive a payload (think “raw bytes”) and confirm with the data provider that the data was successfully received. That’s all the HTTP / 200 response means.

It is the adapter’s job to reference a schema (e.g. GeoEvent Definition) and construct an event record from the payload provided by its associated transport. That’s why you can configure new connectors to, say, receive JSON over TCP – when out-of-the-box there is only a receive delimited Text over TCP – without writing any code. You are just configuring a new inbound connector paring an out-of-the-box transport with an adapter.

The HTTP / 200 returned from an inbound connector, by its transport, does not mean that the inbound connector’s adapter was successfully able to instantiate an event record for processing, that the event record was successfully processed by a GeoEvent Service (configuration errors in a processor might end up throwing exceptions) or that an output was able to disseminate the event record’s data.

- RJ

RahulMetangale2

Hi RJ Sunderman

I went through this entire post, thank you for sharing detailed information!

This is regarding ArcGIS Geoevent Server 10.7.x deployment. Could you please review below diagram and comments and let me know if my understanding is correct.

Below points are based on Geoevent servers multi machine single site deployment.

When external source is sending information to Geoevent server over REST endpoint, my assumption is when one of the geoevent server goes down, other instances will not be able to get the data. We will have to bring external load balancer to address this issue. Load balancer will be responsible for forwarding that request to individual geovent/ArcGIS Server instead of ArcGIS Server site
When Geoevent server is fetching the information from external messaging brokers like ESB, RabbitMQ etc, we don't need load balancer as geoevent server/ArcGIS Server will be able to start the adapter on available instance.
In this deployment (multi instance, single site) although stream service will run on jvm of geoevent server used to publish the stream service. When same geoevent server instance goes down, stream service broadcasts are forwarded to other available GeoEvent Server instances in an ArcGIS Server site.
Mapping client should consume stream service through reverse proxy so that it does not have to worry about specific geoevent server going down.

Thank you,

Rahul

RJSunderman

Hello Rahul -

My apologies for the delayed reply. Generally, yes, the understanding expressed in the points you call out is correct.

>> In a data push scenario, when an external web server, web service, or data provider sends data to GeoEvent Server, the data is sent to a specific resource endpoint via a fully-qualified hostname and port. If this resource is one of three GeoEvent Server instances you have configured – that represents a single point of failure. You will need a load balancer which is smart enough to continue sending data to a specific endpoint as long as that endpoint is available and responding HTTP/200 when data is received. The load balancer will be the solution component responsible for redirection to a different resource / endpoint when the primary receiver is not responding.

>> In a data pull scenario, when a GeoEvent Server inbound connector is polling an external web server or web service, GeoEvent Server (in a "site" deployment) should detect when a machine has left the site's configuration and allow another GeoEvent Server instance to adopt and begin running the inbound connector's polls. This resilience, allowing event record ingest to fail-over to another instance, is one advantage to the "site" approach. You will want to administratively monitor and determine why an instance has failed or left the ArcGIS Server site configuration and confirm that another running instance of GeoEvent Server has adopted the running input. This is one of those "trust but verify" scenarios.

>> Yes, each instance of GeoEvent Server runs within its own JVM, and the web socket used to broadcast data for a stream service is run from within the GeoEvent Server's JVM container (not by ArcGIS Server as a SOC process). The stream service outbound connector implements a fan-out strategy which uses an internal message bus to forward copies of processed event records to other stream service web sockets so that client applications can subscribe to any GeoEvent Server's web socket and get all of the event records regardless of which instance(s) actually processed the event data. You will need to monitor the event record velocity / volume of each subscribing client. Too many clients subscribing to any single web socket instance will reduce data throughput to all subscribing clients.

>> Implementing some sort of reverse proxy to allow web mapping applications and clients to subscribe to stream service web sockets without knowing the specific resource they are connecting to is one way for you to take control of client subscription distribution. The ArcGIS Server stream service, as I understand it, directs client subscription requests to an available server within the site using a round-robin mechanism. But once a subscription connection has been made the client web application is communicating with the web socket GeoEvent Server is running, not with the stream service. The problem here is that if / when a GeoEvent Server instance fails there is no notification to a client application to signal it to unsubscribe and re-subscribe, giving them an opportunity to connect to an available / running GeoEvent Server instance. A brute force work around one of our distributors decided to try was to have their client apps actively, periodically, unsubscribe and re-subscribe. That way, if a client were connected to a "dead" web socket the periodic unsubscribe / re-subscribe would allow automatic recovery and reconnection. This obviously depends on the velocity and volume of event records being broadcast in your solution. A second mitigation the distributor adopted was to train users that if they had any reason to feel that their web map's display was stale to manually refresh the web page which explicitly causes the same unsubscribe / re-subscribe to occur.

I am not a system architect. You will probably want to work with your Esri Technical Advisor or request to speak with a Technical Account Manager through Esri Technical Support for help identifying a resource in Esri Professional Services which can help you with system / solution architecture best practices.

Best Regards --

Eduardo_Fernandez_EsriAu

Hello

The GeoEvent Tutorial says that for high availability, a minimum of three machines and a maximum of five machines are reccomended. I am curious to know why five machines is the maximum?

Cheers Ed

RJSunderman

Hello Eduardo Fernandez –

The reason we recommend a maximum of five machines has more to do with system administrative burden than any hard technical limit in the GeoEvent Server implementation. Inherent administrative challenges with a 'site' deployment (where you deploy multiple ArcGIS Server instances, each running GeoEvent Server, in a single ArcGIS Server site) multiply with the number of machines you add. Also, testing has shown that GeoEvent Server does not scale linearly and you approach the point of diminishing returns when trying to architect a solution with more than five machines.

Hope this information is helpful –
RJ

Eduardo_Fernandez_EsriAu

Thank you RJ, this information is helpful and insightful.

rshihab

Dear RJSunderman,

I have tried to access the tutorial resources but I am getting an error You do not have permissions to access this resource. will you please assist .

thanks

Rejoindre la conversation

Les membres connectés peuvent publier, suivre les mises à jour, et plus encore. Nouveau ici ? Inscrivez-vous gratuitement.

Liens rapides

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.