Number of Services question

[DonaldBrown]

I have 2 ArcGIS Server 10.1 instances running on Linux. Each machine has 2 cores and 8GB. All my services have the settings:
Minimum number of instances per machine: 0
Maximum number of instances per machine: 8
Low Isolation
Instances per process (low isolation only): 8
My question is:
Do these services still consume memory even though they are not being used, and have no running instances?
If so, how is the memory usage spread through a multiple-machine deployment. I am tasked with managing a VERY large number of Services that get relatively little use. I've been having problems that I think are related to memory. Would adding more Server machines to the site help? Do you have any other suggestions for maintaining 200+ Services started on my site?
Thanks!
Don

[KevinHibma]

I want to step through your question 1 part at a time...(apologies if you know these details, they could be helpful to someone else)

If you have your services set to 0 min instances, they'll use 0 memory. That is till the first time they're used.
With your setup of min:0, max:8, if the service never used, it'll never start up, never consume memory.
If its hardly used, then the only time it'll increase an instance is when an instance is busy serving someone else, a 2nd instance will start up. This is true till you hit 8 instances (7 busy instances, one more request, 8th spins up - after that, people wait to get a free instance). Note - this point is for High Isolation.

Now you said you have 2 machines. And it sounds like they participate in the same cluster? So the numbers I threw at you above are all multiplied by 2. When that 1st request is made (assuming instances are at 0), a new instance will spin up on both machines. Remember, the wording here is key:

Minimum number of instances per machine: 0
Maximum number of instances per machine: 8

"per machine", and you have 2 machines. Each instance will consume X* amount of memory.

X* being dependent on the number of layers, features, etc inside the service.

So the above would hold true for high isolation, but you've changed to low isolation.
I'd start with this help topic if you haven't already read it. With low isolation a new instance doesn't get spun up for 1-8. Those max of 8 "instances" are all within 1 actual process. I personally dont have a lot of experience with low isolation map services, so I can't guess at the amount of memory they'd use. A good test would be to start a service in high isolation, with min:8 instances. Check the memory usage of the 8 instances on the machine. Go back, change to low isolation. You'll only have 1 instance, but whats the memory of that? Comparing these 2 might help you further size your machine. (Be sure to read the help topic though to understand the pros/cons of the different isolation methods)

Just my 2cents - I think your min=0 approach is good if you need a large number of seldom used services. The help page linked above also explains time outs. The key point is here:

A third timeout dictates the maximum time an idle instance can be kept running. When services go out of use, they are kept running on the server until another client needs the instance. A running instance that is not in use still consumes some memory on the server. You can minimize your number of running services and therefore conserve memory by shortening this idle timeout, the default of which is 1,800 seconds (30 minutes). The disadvantage of a short idle timeout is that when all running services time out, subsequent clients need to wait for new instances to be created.

If your users can handle the startup time of a 0 instance service, then you can spin services up "on demand" and have them shut down when not in use (well, shut down not long after they've stopped being used).

With all that in mind you'll probably have to do some 1 off scenarios to determine how many services are usually up at a given time. After that, see how much memory they're consuming. Adding another machine to the mix might be helpful (either included in the cluster, or it being its own independent server with X number of dedicated services to host). However if you determine you're running out of RAM, then an alternative might just be to increase RAM on the 2 boxes you already have. Don't get me wrong though, I'm not saying "go buy hardware" - I'm saying do some tests and see if you need to buy hardware. Your user load will probably determine how you need to proceed.

[Anonymous User]

this is a good thread.

I'm reviewing the GIS Wiki, Capacity Tool, System Designer, etc. I want to get my hands on System Monitor, MXDPerfStat  and other tools. Our server has 96 gig ram, hundreds of seldom used services. I already moved those down to 0 instances. 

But for a dozen or so very very big services, with hundreds of layers and hundreds of users (perhaps 10 requests or even 20, at a time at any one moment) on fast hardware and intranet, on 10.2.2 feeding WAB sites as dyn map services.

Question is: what should their max. # instances be?  

I'd read it should be n * 4. where n is cores.  So three times the number of cores. Here you say 2 to 4.  This says 4 - Server Software Performance (CPT Demos) - GIS Wiki | The GIS Encyclopedia  Server Software Performance - GIS Wiki | The GIS Encyclopedia 

I wanted to confirm that is right. But I am simply going to test it immediately, on a few sites, and see what happens. Tackle it empirically to see. I will report my findings back.  

I note the differentiate between batch processes like geocoding locators, and dynamic map services to display vector layers. My use for the services is just to display vector layers, and run Identifies on, so as regular vector dynamic map services to show map layers in viewers.

Also wondering, vs having many mxds and services all of the same layers but slightly differently symboliized, it would seem smarter to have one layer of say, streets. And then link to it from many different webmaps either AGOL or Portal. Symbolize from there. Or definition query.  And make the number of instances huge.  Would that be any better or different with RAM and CPU and enduser experience and performance, vs having many MXDs and publishing a new unique mxd as a service for each time I create a viewer? i.e. Neighborhood Viewer, Parcel Viewer, Streets Viewer, they all have a streets layer in the MXD.  Also, they point to the SDE which I've heard is not optimal. I'm considering replicating to a fGDB and also caching lesser used layers. Those are separate issues.

But for now, I just want to get a handle on # of instances. And other service settings. RAM and networking are no limit. gigE intranet and if we need more cores or RAM we'll get it.   Thanks for any thoughts..