10.1 Memory Issues

[JustinGreco]

I have installed 10.1 onto a new server and have been experiencing issues with memory usage.  Our current 10 installation of ArcGIS Server utilizes about 12 MB of memory, while 10.1 is utilizing about twice the amount of memory with less services running than on our 10 server.  Also the javaw.exe process grows to over 2,000,000 K.  Has anyone else had issues like this with 10.1?  Tech support did not seem aware of this issue.

[SteveTimbrook]

We are also having an issue with the memory growing on certain services.  In our case we are seeing this issue with services that are referencing ECW images. Has anyone else seen this?
Steve

[ToraJohnson]

Help! We're experiencing the same memory issue with zillions of SOC instances and huge Java processes. It's rendered our server unusable. It looks like no one on this thread has found a solution, but I'm posting just in case.

Thanks!

[KeG]

don't have an answer, just suggestions. have you tried Kevin's suggestion:

Just a tip, which may or may not be useful in this situation:
If you goto Task Manager, and under VIEW > Select Columns.
Turn on "Command Line".
In the processes tab you can then see in for each ArcSOC.exe process which service it is running.
The name will be found at DService = "...."
This might be useful in helping to identify which services require more memory than others and planning deployment.

Also, have you tried ramping up the services one at a time to see how that affects things.
and can you align 'heavy' map documents with 'server intense' services.
i think there is also a way to set zero instances for a service, despite the fact that they are all pooled. not sure what zero instances means for a pooled service - i guess maybe it doesn't have an instance until the first call?

[DominicJames]

We are finding that after caching a service, that CPU is 100% - the culprit appears to be .javaw. Restating the service appears to (eventually) solve the problem. This is obviously a bug.

I would be curious to know what settings were changed in the VM environment that solved this problem so we can test it.

[ArtUllman]

I'm going to wake this thread up.

Anybody know what the VM setting to change is?

We had a similar problem.  We originally started out with only 4GB of RAM on a VM, which was not enough.  We requested more memory.  Well, they allocated more virtual memory to the VM, but kept the same physical memory settings (without tell us this).  As a result, the OS thinks it has more memory, but there is a balloon driver in VM that actually steals all of the fake memory so the applications do not get to use any of the memory (which doesn't exist anyway).  We kept seeing memory utilization go up every time we requested more memory, because the balloon driver kept consuming the additional virtual memory.  We thought it was ArcGIS, but was really the VM Settings.  You can see the "private" balloon memory using a tool from sysinternals called RAMMAP.

[DarrenMackiewicz]

No help here Esri????

Did the original person who posted about changing the settings in the VM ever reply with an answer?
I'm currently experiencing this with a 10.1 ArcGIS Server implementation, so this IS still an issue.

[BryanEvans]

Warning: long post!

I wanted to chime in since we've done quite a bit of testing and have noticed a few things that may help some people on this thread.

Oh, and I noticed that stopping the ArcGIS Server and ArcGIS Web Applications Manager brings down memory use from 9Gb out of 10 to 1.6-ish. Turning them back on doesn't bring them right back up, but CPU is close to 100% and memory is creeping up pretty fast...
RJ

The reason this happens is because stopping/starting the ArcGIS Server windows service essentially restarts everything. So the memory drops substantially in a short amount of time because it drops all the ArcSOC.exe's. When the service starts, the CPU jumps to nearly 100% for a bit because it's recreating all those ArcSOC.exe (which correlate to service instances). I've noticed that depending on the number of services you have and minimum instances you have set, this could take some time. To make matters worse - if for whatever reason you have to restart the service mid-day while people are still trying to access it, this process could take longer because it could potentially bump up the number of instances if you have the minimum instances set low.

Help! We're experiencing the same memory issue with zillions of SOC instances and huge Java processes. It's rendered our server unusable. It looks like no one on this thread has found a solution, but I'm posting just in case.

Thanks!

I don't have a direct answer to fix your situation but it provides a good segue into listing what we've learned about ArcGIS Server 10.1. Hopefully it helps you understand the issues you having and how to possibly tackle them.

What we've learned (and I apologize if this somewhat common knowledge):

• ArcGIS Server 10.1 is a complete architecture change/potential rewrite (at least I'm pretty sure). Server 10 running on Windows servers was written in .NET which explains the ability to use local server groups in security, connecting over port 80 without use of the Web Adaptor (because it was an IIS application), DCOM, etc. Server 10.1 looks to be based off Apache Geronimo (a Java based open source application server). That's why you see javaw.exe's (Java worker processes), it has the cluster ability, you need the Web Adaptor (running in IIS) to get Windows Authentication and traffic over port 80, etc.


• The two javaw.exe's you see are Java worker processes like I mentioned. From all my installs it looks like one is dedicated to logging. I don't know the specific use of the other but it seems to be heavily used for some people. I'd encourage people to add the "Command Line" column in Task Manager like khibma mentioned. You should be able to tell from that column if it's the logging process or the other one.


• I haven't done an apples to apples testing of Server 10 to 10.1, but I tend to agree that 10.1 does seem to use more overall memory. I don't know if it's purely the 64-bit aspect or other factors, but I will mention that we've personally noticed that Server 10.1 will increase service instances with nothing more than 1 or 2 requests (and increasing instances means creating a new ArcSOC.exe process which eats up more memory). How we found this was that we published a new service on a server no one was using but me. It was set to min instances of 2 and max of 4. When the service started, it created the 2 instances (I was monitoring via Server Manager). I made one map request to the service, watched the "Instances in use" go to 1 then back to 0, then watched as the min instances increased to 3 even though 2 were already running. I did this again a couple times and in a few minutes it was up to the max of 4 instances. I never caused the instances in use to go over 1, yet the number of instances rose. Maybe this was an odd case, but the moral is watch what your number of running instances are because that is one of the biggest factors in memory use.


• To reduce the number of running instances you have a few options: restart the individual service (which will drop all running ArcSOC.exe's for that service on all machines in the site and start only the min instance on each machine), stop and start each machine in the site (this will drop all ArcSOC.exe's and the javaw.exe that isn't used for logging (note: the ArcGIS Server windows service and the javaw.exe for logging will still be running on the machine) and recreate them at the min instances per machine), or stop and start the ArcGIS Server windows service on each machine (this one will do the same as stopping the machine but also drop the ArcGIS Server.exe and javaw.exe used for logging).


• To give an idea of memory usage for us, we have a Windows Server 2008 R2 VM using Hyper-V3 (says our server admin group) with 4 cores (Intel Xeon @ 3.07 GHz) and 24GB of memory. Right now I counted 86 ArcSOC.exe/running service instances and it's currently using 15.6 GB of memory (65% utilization). I'd say most of our services are medium-sized with some larger-sized services. We also have logging set to a higher output and the javaw.exe for logging is using 1.2 GB of RAM.


• Kind of a general thing I've noticed is memory use is correlated with number of running instances (more instances mean more memory used). Service requests are what impact CPU usage. It can be a tricky thing to calculate out but you'll want an appropriate number of instances to satisfy requests so they don't queue up (causing CPU to get hammered while it renders map requests), but you don't want too many running instances (causing memory use to get hammered). You also have to take into account how long requests to your services take to return. If you have bloated services that take 20 seconds to complete and you get multiple requests, you'll quickly see CPU usage spike up and stay until they can complete. From things I've read, over 85% utilization constantly for either CPU or RAM severely degrades performance. Ideally memory utilization is relatively flat somewhere under 85% and CPU usage goes up and down but never 85%+ for an extended period of time. You'll see CPU usage go to 100% for 5-10 seconds at times but it shouldn't last minutes/hours.


• If you have to use the Web Adaptor for multiple machine sites or for Windows Authentication (IIS only), that's another variable to watch. We've noticed flakiness in the IIS Web Adaptor. Sometimes it just gives out and bouncing IIS doesn't fix it. We restart IIS nightly but sometimes it still has issues. We haven't gotten with ESRI about anything specific about it yet but I've read other forum posts mentioning frustration with the Web Adaptor.

Sorry for the extremely long post but hopefully it helps some people understand what's going on with Server 10.1 and how to configure things to get better performance out of it.

[bbieb]

Just to add more memory information.
We have 2 identical VM machines, one running 10.0 SP2 and the other is running 10.2.  Both machines have the same services running...which equates to about 100 ArcSoc.exes on each machine.  On the 10.0 machine, the ArcSoc.exes average 77k and on the 10.2 machine, the ArcSOC.exes average 132k.

I've been talking to support about the increased memory usage and their response was that it is documented in the help.  I was pointed to documentation for 10.0, http://help.arcgis.com/en/systemrequirements/index.html#/ArcGIS_Server_10_System_Requirements/015100... and to the documentation of for 10.1, http://resources.arcgis.com/en/help/system-requirements/10.1/index.html#//015100000072000000 .

If you read very closely, the specs from the development environment changed from 2GB to 4GB.  I guess from that you are suppose to infer about the increased memory usage between versions.

I was also given this bug number which was declined.  NIM092919

This doesn't solve anyone's problems but just confirms what everyone is seeing.  And you will see the same memory increase in 10.2.
brian

[ScottPrindle]

As stated above, it is very helpful to enable the Command Line column in Task Manager. This will show more information related to each process. The ArcSOC.exe processes are all associated with your services, and this is revealed through the Command Line column. This can help put a finger on an individual service that may be consuming an extraordinary amount of memory.

The javaw.exe processes do a vast majority of the underlying ArcGIS Server work. The one that has "logging" in the Command Line is not necessarily solely for logging. It is likely the underlying java process that is managing and running your server.

Another place to look to ensure you aren't running out of system resources is the System Commit under the Performance tab in Task Manager. If this ratio is reaching 1 (ex. 15/16GB), then you are running out of virtual memory on the server. To get a better idea of the total memory consumption of each process, turn on the Commit Size column for the Processes tab. This will be a higher value than the Memory (Private Working Set). If you are seeing any performance issues and the System Commit is maxing out then you may need to provide more virtual memory for the machine. This can be done through Advanced System Properties>Performance Settings>Advanced>Change Virtual Memory. Ideally the virtual memory should be managed by the system, but it can also be increased to a more acceptable level.

Separate from the system properties themselves, service properties can also be modified to better suit the system they exist on. If a map service is rarely used, its minimum number of instances can be lowered to 0 or 1. Many services will be just fine with a maximum of 2-4 instances as well. If the server is a much larger deployment, then it would be prudent to use the Capacity Planning Tool:
http://wiki.gis.com/wiki/index.php/Capacity_Planning_Tool

Tuning and configuring services:
http://resources.arcgis.com/en/help/main/10.2/0154/015400000343000000.htm

[MikeHargreaves]

We have been having some similar issues.  This morning, one of the javew.exe processes were using all of the CPU.  We stopped the process, and now it seems to be fine.  We've upped the RAM on the VM, and have had less issues like this.  Before, we would see memory creep up, and then CPU max out, but it has not been a problem for a month or so until this morning.  Any ideas what might have happened this time?