GeoEvent and High Availibility

[AzinSharaf]

This would be our Portal and ArcGIS Server high availability deployment.

We want to add a HA GeoEvent Server to this architecture. What would be the best solution based on the following limitations/recommendations?

1- Don't use multi-machine sites with GeoEvent

2- Don't install GeoEvent in your base GIS site

If we create two independent GeoEvent site, we should sync them manually? Any better solution?

[RJSunderman]

Hello Azin -

The ArcGIS Server documentation includes reference architectures for deploying additional server roles (such as GeoEvent Server) to your ArcGIS Enterprise.  The terms I'm using here are consistent with the 10.5.x product release. There are reference architectures for high availability as well as for GeoEvent Server.

Based on the icons you included in your system architecture illustration, I might assume you've seen these references - but I wanted to make sure.

I would point out that the reference architecture for GeoEvent Server which depicts a horizontal scale-out deliberately has white space between the machines:

The white space is intended to depict your first stipulation -- that you do not 'Join Existing Site' when installing the GIS Server component of ArcGIS Server with the intention of later installing GeoEvent Server. Each of the three GeoEvent Server machines depicted above exist in their own site. They do not know anything about one another. This is to prevent leveraging a concept we refer to as "GeoEvent Clustering" which we introduced with the 10.3 release. Changes to address some issues were included with the 10.4 release. We eventually realized that the concept was fundamentally not working as we intended, and the advice now for releases 10.3 / 10.4 / 10.5 is to not have more than one GIS Server in a site when GeoEvent Server is installed.

We are working on developing a "Gateway" component for 10.6 with the intent that GeoEvent Server will again offer some resiliency and improved scalability. However, until then, your challenge is to figure out how to configure an external component to handle event distribution to two or more instances of GeoEvent Server which do not know anything about one another.

We prepared a GeoEvent Server Resiliency tutorial which proposes using Kafka as an external message broker. Kafka provides load balancing to your real-time event message ingest by holding messages in a queue until consumers (individual GeoEvent Server instances) fetch them at the rate they are able to process them. My terminology may be off; I would recommend you refer to Kafka's documentation as you read through the GeoEvent Server Resiliency tutorial.

I hope this information is helpful --

RJ

[AzinSharaf]

Thanks rsunderman-esristaff

[HarroldSompotan]

Hi RJ,

Now that Arcgis for Server and Geoevent Server 10.6 is available, and the Gateway components  are in place, are we still encouraging other to go with a siloed Arcgis Geoevent Server instead of Arcgis Geoevent Server in a multimachine site? 

Currently the documentation at 10.6 still mentions below:

"It is NOT recommended to use multi-machine sites with ArcGIS GeoEvent Server. Instead, you should configure multiple single-machine sites to dedicate ArcGIS GeoEvent Server machine resources to specific real-time data streams:"

Deployment patterns for ArcGIS Enterprise—ArcGIS Enterprise | ArcGIS Enterprise 

[RJSunderman]

Yes, with the introduction of the GeoEvent Gateway, system architects again have the option of deploying GeoEvent Server with multiple machines configured as part of a single ArcGIS Server site. The recommendation is still that your ArcGIS Enterprise be deployed separately allowing you the flexibility to scale the GIS Server component for map and feature services.

The only real change to the diagram would be to de-emphasize the white space between GeoEvent Server instances:

Above, I'm illustrating that a GeoEvent Gateway, running on each ArcGIS Server machine, will coordinate through the ArcGIS Server site dedicated for real-time event processing to distribute events processing across the three machines.

There are still a number of questions to prove out here. What if a solution were being designed which had several, independent, inbound feeds whose velocity and/or event volume were each nominal (no more than 1,000 or 2,000 events per second)? Such a use case wouldn't require an event distribution mechanism to leverage several machines. It might be better, in this case, to follow the 10.5 / 10.5.1 pattern of deploying independent instances of GeoEvent Server in separate sites, dedicating instances to handle one or two independent inbound data streams. The above single-site, multi-machine approach would be recommended only if you were deploying 10.6 and had a single inbound event stream whose velocity / event volume were too much for a single server to handle.

A tutorial is being prepared for system architects and solution engineers which will identify recommended best practices for product deployment in a multi-machine configuration. The tutorial should be available by the Developer Summit, March 6 - 9 2018. Please reach out to Greg Tieman‌ with questions on this. You can read more about the GeoEvent Gateway in the on-line help which was pushed out 9-January:  What's New at 10.6 GeoEvent Server

(Internationalization of the on-line help is still pending as I post this reply...)

- RJ

[JohnDye2]

Sorry, I'm a little thick sometimes. 

Just for my sanity:

• If your solution has multiple inbound feeds with an event velocity and/or volume that did not exceed 2,000 events per second, it is recommended to deploy multiple single-machine geoevent sites and dedicate each to a one or two inbound feeds.
• If your solution includes inbound feeds where the event velocity of volume does exceed 2,000 events per second, then an event distribution mechanism is likely needed and as such, a multi-machine geoevent site is recommended so that the geoevent-gateway can distribute events among the machines.

Is that correct?

[RJSunderman]

John Dye‌ ... you have correctly interpreted the recommendation.

You probably want to defer production engagements you think require GeoEvent multi-machine functionality and continue to deploy GeoEvent Server in separate ArcGIS Server sites. There's still work we need to complete to document best practices for you. Also, there are some issues we are investigating which tend to manifest with deliberate fault injection (like abruptly powering down a machine or severing its network connection). There will likely be a patch for 10.6 for multi-machine deployments ... stay tuned.

- RJ

[JohnDye2]

Thanks RJ. I suppose the one question your responses then raises is whether or not there will be a streamlined process for moving from a GeoEvent single-machine-multi-site configuration to a multi-machine-single-site configuration once this is all said and done.

It sure would be a headache to go with the former only to have to undertake major efforts to move to the latter at say, 10.6.1 or 10.7

[RJSunderman]

I normally prefer exporting my GeoEvent configuration to an XML file, then uninstalling GeoEvent Server and installing the newer release (vs. following an upgrade-in-place workflow). Product upgrades make more sense for ArcGIS Server and Portal for ArcGIS where exporting and importing your site configuration can be more difficult. For GeoEvent Server, it's really easy to snapshot the configuration as XML and then selectively import just the inputs, outputs, GeoEvent Services (etc.) that you know you're using - or import the entire configuration.

In the case you're describing, where you want to scale your deployment from a single-machine architecture to a multi-machine architecture ... I would definitely recommend you export the GeoEvent configuration, perform whatever s/w upgrades are needed to move to 10.6, establish your ArcGIS for Server site with its multiple machines and multiple instances of GeoEvent Server, then import your GeoEvent configuration once, from the XML. I would then test to make sure that all GeoEvent Server instances 1) receive the configured elements; and 2) event processing is distributed across the machines in the site when event data is sent to any one machine's input.

- RJ

[rshihab]

Dear RJSunderman,

I have tried to access the tutorial resources but I am getting an error You do not have permissions to access this resource. will you please assist .

thanks