nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

What is the fastest way to add a new machine to an ArcGIS Server site

berniejconnors

We recently had some heavy load on our ArcGIS Server site. We have three machines in the site and we considered adding a fourth machine to handle the increased load. I thought I might ba able to clone one of our our existing ArcGIS Server VMs to speed up the process of adding a new machine. However, I got this information from Esri Canada Support:

"To add a new machine to the site: Cloning a VM is not supported and we don't recommend using it. You can create a new VM, install ArcGIS Server and make it join the existing site: https://enterprise.arcgis.com/en/server/10.9.1/deploy/windows/join-a-machine-to-a-site.htm"

So what are your suggestions for the fastest way to spin up a new VM with ArcGIS Server? We currently install ArcGIS server via the setup GUI and it is not a fast process.

Thanks,

Bernie.

P.S. Our heavy load issues have been resolved and we are not in a crisis situation. I'm just looking for some guidance for the future.

Encuentra más publicaciones etiquetadas con

Accepted answers

JoshuaBixby

The largest multi-machine ArcGIS Server site I have ever deployed and managed was 6 machines, and I have deployed dozens of smaller multi-machine sites. The 6-machine site was always the least stable of all multi-machine sites, and I didn't realize how much horizontal-scaling was a factor until we added more resources to the machines and dropped from 6 machines to 4 and then eventually 3. I have come to strongly discourage scaling horizontally above 3 machines, and I encourage people to look at vertically scaling instead by adding resources to machines.

Although Esri doesn't state there is a maximum to how many machines can be in a AGS site, there is definitely a practical limit, and I have found it is lower than one would think. The main issue with horizontal scaling becomes the shared working directories, especially the shared configuration store. The IOPS for adding a new machine to a multi-machine site have always seemed to scale linearly, which on the surface sounds OK when thinking of how algorithms scale with data set size. However, when the IOPS are as high as ArcGIS Server's shared configuration store, horizontal scaling really starts to chew up IOPS. Vertically scaling generates no extra IOPS except for those additional services you are running now that you have more resources.

Without knowing more specifics of your service types and uses, it is hard to offer any specific suggestions. Is there a reason you can't vertically scale?

All comments

CodyPatterson

Hey @berniejconnors

I would consider checking out the Chef Cookbook here that Esri has provided:

https://github.com/esri/arcgis-cookbook

This provides a better automated way for setup, and may end up helping a ton when it comes to the automation and ease to setup!

Cody

JoshuaBixby

berniejconnors

@JoshuaBixby that's great feedback from someone with experience running 6 machines in one site.

Scott_Tansley

I love the explanation by @JoshuaBixby and concur wholeheartedly. I personally recommend my clients to 'abstract' out workflows on to different servers and server sites. Hosting machine just for hosting, server for traditional workflows, imagery for image data which is slower to read. There are other use cases as well. By having a single machine, you have observability of that machine and as Joshua says, you can scale that machine up. If there is a genuine need for High Availability, then push out to two machines on the site.

Ultimately following the concepts of the Well Architected Framework has more to offer than just adding more machines to the same site and hoping it will work.

I also agree with not cloning. You can stand up an ArcGIS Server in pretty short order these days, especially if it's not federated (I know yours aren't). Then join it to the site if you need to. There are too many internal configurations that hold references to the machine name, to start cloning machines then making changes after. The clean build approach always has the fastest outcome.

Final though is that back in the day (when we ring-fenced ArcGIS Servers for a 'product' that hardly changed) you could create images of a machine and effectively put it in read-only mode. Everything would be contained on that's machine image, e.g. file geodatabases - this mitigates the I/O issue that Joshua raised. All the machines are 100% equal, but separate - to be pedantic, they are exact copies of each other - but they are not behaving as a site or cluster.

You would need to configure a third-party load balancer, which would round robin, so machine 1, machine 2, machine 1, machine 2. If you brought machine 3 up, then you added it to the load balancer.

It's quite an inflexible model, but it works beautifully if you can tolerate the constraints, like you can't really republish without an outage...

I haven't seen this approach taken in a long time, but outside of an 'Enterprise' it's still theoretically possible.

berniejconnors

When two experts agree it really increases confidence in the solution. Thank you very much @JoshuaBixby and @Scott_Tansley for freely sharing your expert opinions.

Bernie.

JoshuaBixby

Another aspect of why vertical scaling makes sense more than horizontal is licensing. I have always worked in large government organizations that deploy lots of what I refer to as "big brother" software, i.e., configuration management software, security software, monitoring software, etc.... Don't get me wrong, these types of applications are needed in today's environments whether they are small business or large government. As needed as they are, they consume resources and increase the amount of resources a machine needs compared to an idealized, bare-bones OS.

For a previous organization I worked for, the rule of thumb was the first 2 vCPU were basically throw-away vCPU to support running all the big-brother software. This was more to do with poor management of the software by operations teams than the software itself. So, if capacity planning shows you would want 4 vCPU for a VM, we always spec'ed 6 to ensure the application would always have roughly 4 vCPU worth of capacity available. ArcGIS Server (not Portal, but Server itself) is licensed per core. Unlike some enterprise software like SQL Server where the cores can be set in the application, ArcGIS Server licenses all the cores on a machine.

For situations like the one I describe where a certain number of vCPUs are set aside for other operations software on the machine, and yet those vCPU are still licensed for ArcGIS Server, horizontal scaling drives up licensing costs much quicker than vertical licensing because each new machine has those marginally used but still licensed vCPUs whereas adding more cores to an existing machine allows having the application use all those new cores.

Regarding @Scott_Tansley comment about segmenting workflows into other ArcGIS Server sites, I agree, at least for medium to large customers. Each type of GIS service has its own unique patterns of use, both from the end-user consuming the service to the service consuming machine resources, and mixing them in large numbers often results in resource utilization and contention issues and capacity planning headaches.

valenj88

Anecdotally: I admin for a medium-size city government, we had been vertically scaling and hitting a lot of bottlenecks with concurrent service loads (think lots of users loading heavy portal maps all at once). Adding just a single load-balancing server machine (using a fresh server install and "joining site") practically alleviated all of our slow-downs and timeouts immediately. Cannot recommend enough. I guess the caveats being what the more experienced guys have described above.

Just make sure you follow the ESRI documentation for adding a load balancing server - they basically act as a mirror image of your main hosting server so need to be provisioned exactly the same (hardware-wise), and have access to the config and data stores of the main site. And if you are going for High Availability, the stores need to live on a separate file server (we do not need HA so I left the stores on the main hosting machine).

JoshuaBixby

If you run into issues with vertical scaling, one area I have seen people run into problems is managing memory, both physical and virtual, at the OS level. ArcSOC processes have a surprisingly large virtual memory footprint relative to their physical memory use, and when scaling RAM up to accommodate more services, I have seen virtual memory sizing and use become a problem. Without knowing a lot more details about the application and OS configuration and spec's, I can't really offer any specifics.

Únete a la Conversación

Los miembros registrados pueden publicar, seguir actualizaciones y más. ¿Nuevo aquí? Regístrate gratis.

Enlaces Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.