Struggling with ideas on setting up and configuring ArcGIS Server in a highly available environment. Suppose it's worth mentioning that I'm struggling with the technical details behind the concept.
In my setup, I have two ArcGIS Servers set up on two separate machines, clustered together, with web adaptor installed on server 1. The web adaptor sees both no problem, as I expected since they're both part of the same site and same cluster. On a third, virtual machine, is our third-part load balancer, and the server site is set up with a geodatabase residing on a fourth server registered as a data store.
In the process of testing it, I realized that if I shut down a service on one server, the corresponding service on the other site also gets shut down. This is not what I wanted to happen, and runs counter to the reason why I and others are standing this up in the first place. What am I missing here that is causing this?
I was also reading through other threads in trying to find my answer and noticed that those who have a setup similar to mine usually have their web adaptor on the same machine as their third party load balancer. Is my having the WA on the same machine as one of my ArcGIS servers going to have any adverse effects down the road?
Since the two machines part part of the same site, any administrative actions performed on one machine, (ex. shutting down a service), are made on the other machine(s). If you want completely distinct Server sites so that type of administrative action does not affect the other machine(s), you'll need to set up an active/active deployment:
Single-machine high-availability (active-active) deployment—ArcGIS Server (Windows) Installation Gui...
This means you have two sites, each identical to each other. You'll need to publish services to both environments, and stopping a service on one site will not stop the service on the other. The thing to be aware of, though, is the load balancer has no awareness about services. They can typically only check the health of the site through the health check URL:
Health Check—ArcGIS REST API: Services Directory | ArcGIS for Developers
If you stop a service on one machine and not the other, but the load balancer is still balancing load between the two machines, then alternate requests will fail as the service won't be returned from one of the machines.
If you have a whole Server go down, then load balancing, (regardless of active/active single machines or multi-machine Server sites) will continue to work as long as the component sending traffic to the machines, (web adaptor or load balancer) is aware of the health of the Server sites, (see health check URL above).
I think this is the direction we're going to go with this one. It does complicate the configuration of the load balancer, but it seems that it's capable of producing the outcomes I'd like to see.