10.1 SP1 Memory Usage

[ronaldmcfarlane]

We have a dedicated server running ArcGIS Server 10.1 and MS Server 2008 R2 with 20 GB of system memory. We just upgraded from 10.0 to 10.1 ArcGIS Server and our available system memory has shrunk dramatically.  Running 10.0, only about 4.5 GB of system resources were consumed; now with 10.1 SP 1, 15+ GB of resources are regularly consumed.  With the ArcGIS Server service shutdown 2.5 GB are consumed, as soon as it is restarted it ramps back up to consuming 70% to 80%.  There appears to be a ArcSOC.exe for each map/image/feature service published with each one consuming ~80 MB of memory.  The size consumed does not seem to coincide with data content, one ArcSOC.exe (map service, with 1 point) is 88 MB while another is 75 MB (map service with 5000+ points).  Is there a good way to manage the memory consumption?  If I add more memory will the services bloat to consume the additional resources and still leave me only ~25% free?

[KevinHibma]

How many services do you have running?

Could I infer that 15 gigs (15000MB) / 80MB avg size = ~188 services? (give or take)

Do you have any outliers? A few that are 500+MBs? Or is every single one "small"?

Since 10 and 10.1 are very different, you can't say that Map Service A on 10.0, x-MB == Map Service A on 10.1, x-MB

[ronaldmcfarlane]

How many services do you have running?

Could I infer that 15 gigs (15000MB) / 80MB avg size = ~188 services? (give or take)

Do you have any outliers? A few that are 500+MBs? Or is every single one "small"?

Since 10 and 10.1 are very different, you can't say that Map Service A on 10.0, x-MB == Map Service A on 10.1, x-MB

I have 136 map/feature services, 5 image services, 2 system and 8 utility; plus javaw.exe (log) at 450-1000 MB & javaw.exe at ~220 MB and ArcGIS Server at 110 MB.  With ArcGIS Server off - 1.65 GB, with it on 16.1 GB.  All the Map/Feature/Image services are between >80 and <90 MB.

[KevinHibma]

Well...thats certainly a lot of services. The amount of memory they use depends greatly on what they're hosting.
Simply based on the numbers and the typical behavior of a map service, it sounds right to me.
In my experience an instance will not use more memory because there is more system memory available. (I run 2 machines, 1 with 4gig RAM and another with 8gig. Services behave the same on each machine)

My question is : are you running out of system resources or do you have something else besides ArcGIS Server on the machine that needs RAM? Or are you more concerned that you need more RAM to support these instances?

I'm not saying "dont do anything", but in my opinion if you're at 75% RAM and during normal day to day use it never hits 99%, then its ok. Or if you have services which rarely to never get used, you could set their Min Instances to 0. This means that the service is available, but no instances are up and ready, thus no memory footprint. The first user to hit that would incur the wait of spin up.

These are just some personal thoughts, hopefully some other forum members with more "worldly" experiences can give their 2 cents.
There is also a wealth of information in the system design wiki: http://www.wiki.gis.com/wiki/index.php/System_Design_Strategies

[ChristopherRoth]

I have 136 map/feature services, 5 image services, 2 system and 8 utility; plus javaw.exe (log) at 450-1000 MB & javaw.exe at ~220 MB and ArcGIS Server at 110 MB. With ArcGIS Server off - 1.65 GB, with it on 16.1 GB. All the Map/Feature/Image services are between >80 and <90 MB.

I nearly choked on my coffee when I read that figure. That is a lot of services by any measure. It would be very reasonable to assume that this number of services would be consuming 15GB of memory.

In terms of tweaks you could go with a more 'on-demand' approach which all of your services having a minimum number of instances set to zero. This would reduce overal memory usage as map/feature services which are not being used will 'go to sleep' and no idle ArcSOC.exe will be sitting around eating memory. Their will be a small lag upon first request of any of these services to spin up a SOC but thats a small price to pay to possibility save GBs of memory.

I have a similar question as khibma, are there other reasons you need to address memory usage on this sytem? High memory usage like this is not inherently bad. The applications using memory are doing so for good reasons and are ready to respond to requests. In my experience this usage will become a problem as you choke out other services and force Windows to aggressively manage memory, such that it has to free some up for other high priority applications or OS services.

Considering your base load is 75% utilization at services at idle state, it makes me worried what will happen when services started getting used and concurrent user sessions start up against those services as memory usage will increase.

Is this server a dedicated host or VM? I'm asking because this approach puts you at huge risk. How many people will be affected if that one server goes offline or has a hardware failure?

[JonathanQuinn]

Going off on the posts above, setting the minimum number of instances on services that aren't used as often will definitely conserve some resources on your server, as a SOC processes won't spin up until it's requested.  The other thing you can do is look into using low isolation for some services.  This will allow multiple requests to use one SOC process so it doesn't spin up another instance when the service is requested.  The only drawback is if two users are using the same SOC process, if that process crashes, both users will be affected.  With high isolation, both users will have a dedicated SOC process.