topic Re: ArcGIS Server multiple-machine deployment Experiences in ArcGIS Enterprise Questions

ArcGIS Server multiple-machine deployment Experiences

ScottCorwin — Fri, 26 Apr 2024 21:36:33 GMT

Hi, looking for feed back on going from an singe ArcGIS GIS Server site to a multi-machine site. Technically it's already a multi-machine site with ArcGIS Image Server on one machine and ArcGIS GIS Server on the other, both are federated. We are running around 150-160 services utilizing over 80% of our memory sometimes spiking into the 90's when someone starts an edit session in portal. So one potential solution would be to add another ArcGIS GIS Server. One specific question I do have, is how does this get managed? Do I as the admin get to decide GP services on machine A and pooled instances on machine B, or does some aspect of the Server software manage this without user input?

Re: ArcGIS Server multiple-machine deployment Experiences

MichaelJenkins — Fri, 26 Apr 2024 22:05:02 GMT

When you have multiple servers joined as a single "site", all of the services are managed the same for both servers. In fact you can manage the services from ArcGIS Server Manager connected to either machine.

You can still set service properties, but those properties will apply to both machines. For example, if you use a dedicated instance and set it to a maximum of 5, that is 5 per server so you are really getting a maximum of 10 instances. If you turn off a service, it is turned off on both machines.

You cannot assign services to one machine or the other, though. To do that you would just federate another server to the Portal and then when you publish a service you can choose which server it will run on.

One other consideration is that both machines must access the same \config-store and \arcgisserver directories, so you would either set those up on a 3rd server or some sort of network file share, or you would put them on one of the two AGS servers and share it so the other mahine can access it too.

Re: ArcGIS Server multiple-machine deployment Experiences

BillFox — Fri, 26 Apr 2024 22:47:40 GMT

if you have the resources, leave what you have as-is while you build up you new HA site from scratch

that will let you go through all the steps without potentially breaking anything or everything

jump in here: https://enterprise.arcgis.com/en/portal/11.1/administer/windows/high-availability-in-arcgis-enterprise.htm

Re: ArcGIS Server multiple-machine deployment Experiences

ScottCorwin — Fri, 26 Apr 2024 22:56:13 GMT

Thank you. If instances are pooled, now 10 avalible in your example then does it follow that requests are then divided amongst the machines? Or is this all more about redundancy should a server die, then you still have 5 instances avalible and perhaps even an active request?

Adding and federating a new server should be straight forward, were already setup for it on the directory and config store (yay), and following review of our monitor logs can see what services should on which service etc.

Thanks Again, hope the shows are good!

Re: ArcGIS Server multiple-machine deployment Experiences

MichaelJenkins — Fri, 26 Apr 2024 23:17:53 GMT

That's right, if two machines are joined as a site then the requests are divided between the two servers. If you look at the log files you can tell which server is providing the response - it is listed in the machine name column. But you can't control which server will respond for individual requests or for individual services.

This is sometimes called "Scaling out", as opposed to "Scaling up", which would be just adding more memory to your single machine.

Federating a new server and assigning specific services to it is a valid approach, too, and would be called "workload separation"

Re: ArcGIS Server multiple-machine deployment Experiences

jcarlson — Fri, 26 Apr 2024 23:45:13 GMT

We've got close to the same number of services, but our memory hangs out around 50% and rarely spikes. You may not actually need another server. Make sure your published services are taking advantage of the shared instance pool. I would recommend avoiding dedicated services entirely unless it's for a mission-critical service. I suppose that could be the case for all your services.

Re: ArcGIS Server multiple-machine deployment Experiences

ScottCorwin — Mon, 29 Apr 2024 20:44:55 GMT

Josh, thank you for the reference, it's always helpful to see how other sites run, how much memory does your server have? We are heavy on dedicated not so much in shared.

Re: ArcGIS Server multiple-machine deployment Experiences

ScottCorwin — Mon, 29 Apr 2024 20:57:25 GMT

Thank you, I like that term, "scaling out", and it looks to provide what we may need and something I will test.

Follow up question about shared directories for the config store/server directories the documentation says;

"Server performance can suffer when shared locations are used for multi-machine site directories and data, owing to several factors — network bandwidth and stability, opportunistic locking, and network drive performance among them. Use of server directories and data in shared network locations can also negatively affect performance of services under heavy load."

The system has been running with share config-store and server directories since setup, I inherited IMO a well planned system, are you aware of performance issues and how they may manifest? I'm not happy with our memory usage and not entirely convinced it's based on the number of services(long long story)

Re: ArcGIS Server multiple-machine deployment Experiences

ScottCorwin — Mon, 29 Apr 2024 20:58:57 GMT

Thank you Bill, had not really looked that way yet. We are in the process of creating a dev site/clone of our production,. we could certainly test this idea here to.

Re: ArcGIS Server multiple-machine deployment Experiences

MichaelJenkins — Mon, 29 Apr 2024 21:43:26 GMT

I can't speak to what the documentation is warning about, but...

If you want to gain more insight into memory usage by services, open Task Manager and click on the "Details" tab.

You will see dozens of ArcSoc.exe processes listed and their memory usage. Each one is a "instance" of a map service. Task manager has some hidden columns with good info. Right click on a column name and you should see "Select columns", then find and select "Command line"

Now you will have a column that tells you exactly which map service each ArcSOC is related to. Look for "Dservice=mapservice name.MapServer"

With the number of services you are working with, I would go for 64GB off RAM on that machine. 32GB would likley not be enough. Curious what JCarlson thinks about that.

Re: ArcGIS Server multiple-machine deployment Experiences

jcarlson — Tue, 30 Apr 2024 00:25:56 GMT

I had to double check this, but it's true: we're doing it on 16 GB RAM! I think 32 would be nice, but it hasn't been necessary for day-to-day operations. But we do nearly everything on shared instances, and the two dedicated services have a minimum instance setting of zero, so that those processes exit entirely when not in use.

EDIT: I also double-checked our service count, and we have over 250 services running on shared instances. So, I think it's working out pretty well.

Re: ArcGIS Server multiple-machine deployment Experiences

jcarlson — Tue, 30 Apr 2024 00:24:52 GMT

Posted up above, but we run all our stuff on a machine with 16 GB RAM. I'd consider 32 in a few years, maybe, but I can't see ever needing more than that. But as I said, we do shared instances whenever possible. We're running over 250 services, too!

Great tip about the command line column, that's awesome!

Re: ArcGIS Server multiple-machine deployment Experiences

ScottCorwin — Tue, 30 Apr 2024 15:12:52 GMT

You can also export the information from task manager, I've used this before, although I think I could have done much the same in Monitor.

Caveat, I found this information here: https://superuser.com/questions/415360/how-do-i-find-out-command-line-arguments-of-a-running-program ran it as is without much research, it worked, but always risk when you do such. 🙂

WMIC /OUTPUT:C:\Process.txt path win32_process get Caption,Processid,Commandline

Re: ArcGIS Server multiple-machine deployment Experiences

ScottCorwin — Tue, 30 Apr 2024 15:27:59 GMT

That's impressive! Is startup delay even perceptible on the dedicated services?

Re: ArcGIS Server multiple-machine deployment Experiences

jcarlson — Wed, 01 May 2024 03:30:11 GMT

No, not really. We only have two dedicated services, both are parcel fabrics, and both are only used internally. Since we work with the fabrics in Pro, the service is usually on and available before Pro has finished loading the project.

Re: ArcGIS Server multiple-machine deployment Experiences

JimCarmichael1 — Wed, 15 May 2024 02:16:44 GMT

When you say you have ArcGIS Image Server on one machine and it’s federated, is it a separate GIS server site or did you end up joining it to your existing ArcGIS server site. We are thinking of deploying an Image Server primarily for image hosting, but I’m confused during setup if I need to join it to the existing GIS Server site or if it’s possible to just create a new site, federate it to our portal, publish image services to it, and then use a web adaptor to access the REST of that Image Server site?

Re: ArcGIS Server multiple-machine deployment Experiences

ScottCorwin — Wed, 22 May 2024 21:59:47 GMT

Our image server is it's own server site. So you are on target for the remainder of your idea, stand up the image server configure the webadaptor and then federate it to portal.

Re: ArcGIS Server multiple-machine deployment Experiences

MichaelJenkins — Wed, 22 May 2024 23:01:33 GMT

You would not want to join it to your existing ArcGIS Server site for the reasons discussed earlier in this thread -- if joined then both machines would split all of the work. Keep them separated.