topic Re: 10.1 Memory Issues in ArcGIS Enterprise Questions

10.1 Memory Issues

JustinGreco — Wed, 05 Sep 2012 11:35:56 GMT

I have installed 10.1 onto a new server and have been experiencing issues with memory usage. Our current 10 installation of ArcGIS Server utilizes about 12 MB of memory, while 10.1 is utilizing about twice the amount of memory with less services running than on our 10 server. Also the javaw.exe process grows to over 2,000,000 K. Has anyone else had issues like this with 10.1? Tech support did not seem aware of this issue.

Re: 10.1 Memory Issues

KevinGooss — Wed, 05 Sep 2012 12:42:40 GMT

i have two javaw.exe processes - one is at 191 and the other is at 278,000K.
I also have 14 ArcSOC.exe processes at the moment, each of those is about 100,000K average.
This is a new box and if we add map services at the rate we normally do we are quickly going to be out of memory.

Re: 10.1 Memory Issues

KevinHibma — Wed, 05 Sep 2012 14:16:49 GMT

I'm not 100% sure of this, but I think one of the javaw.exe is for logging.
Do you have logging set to a high level? If you set it back to a low level (like SEVERE), and re-start ArcGIS Server, it shouldnt be as active.

Re: 10.1 Memory Issues

JustinGreco — Wed, 05 Sep 2012 15:24:50 GMT

Kevin,

I have the log level set at warning and just switched it to severe, but the javaw is still using over 1,000,000 KB. The other issue is the amount of memory ArcGIS Server is using at 10.1, almost twice was much as 10. Is that to be expected at 10.1? I also notice that when I stop the ArcGIS Server service, the memory usage does not drop at all.

Re: 10.1 Memory Issues

KevinHibma — Wed, 05 Sep 2012 15:39:44 GMT

If you've stopped and re-started the ArcGISServer service in the Services panel, and memory didnt change, then the process sounds either hung or orphaned.
When you restart they should all disappear and then come back fresh.

After re-starting my server I have:

2 javaw.exe

1 arcgisserver.exe

X # of ArcSOC.exe

I'd expect you to have something similar.

I cant give you a yes/no answer on how much memory should be used. I don't concentrate on the core architecture and wouldnt want to give you a wrong answer.
Just my impression though, 1 gig (1,000,000KB) sounds high for the javaw.exe process, but without knowing exactly what your server is doing, # of requests, # of services, etc - its hard to say if thats a problem. The biggest change which I'm sure you're aware of is 10.1 being 64bit, so each process can potentially consume more memory than 10.0 -32bit server.

Just a tip, which may or may not be useful in this situation:
If you goto Task Manager, and under VIEW > Select Columns.
Turn on "Command Line".
In the processes tab you can then see in for each ArcSOC.exe process which service it is running.
The name will be found at DService = "...."
This might be useful in helping to identify which services require more memory than others and planning deployment.

Re: 10.1 Memory Issues

JustinGreco — Wed, 19 Sep 2012 00:52:30 GMT

Turns out it wasn't an ArcGIS Server issue, but a setting that needed to be changed in our VM software after increasing the amount of RAM.

Re: 10.1 Memory Issues

ChrisOrpurt — Mon, 01 Oct 2012 12:04:38 GMT

Turns out it wasn't an ArcGIS Server issue, but a setting that needed to be changed in our VM software after increasing the amount of RAM.

Can you post what that setting was?

Re: 10.1 Memory Issues

Jay_Gregory — Fri, 19 Oct 2012 12:17:23 GMT

I'm having similar memory problems with a new install of Server 10.1 - any idea what VM setting you changed?

Re: 10.1 Memory Issues

GeraldLee__GISP — Fri, 19 Oct 2012 13:34:53 GMT

Count me in with this same issue--my 10.1 memory usage is about 2x as same 10.0 services, stopping services doesnt seem to relieve CPU, etc..i also am using VMware..on windows server 2008 R2. My IT said i must be hallucinating...glad to know its not just me.. 🙂

Re: 10.1 Memory Issues

EricKriener — Thu, 28 Feb 2013 14:34:38 GMT

I'm having similar memory problems with a new install of Server 10.1 - any idea what VM setting you changed?

I'm going to wake this thread up.

Anybody know what the VM setting to change is?

Re: 10.1 Memory Issues

AndrewWharton — Thu, 07 Mar 2013 11:43:50 GMT

I just encountered the same issue with one instance of javaw.exe using over 3,200,000kb. Two other instances were in that 200,000kb range. We are using 10.1 but aren't using VM software, so it looks like an issue or setting that affects more than just VM.

Re: 10.1 Memory Issues

RobertWeise — Thu, 11 Apr 2013 21:16:57 GMT

And same here: no VM, memory at 90% capacity, and often CPU usage peaking towards 100%. Stopping services does not make a difference.
RJ

Re: 10.1 Memory Issues

RobertWeise — Thu, 11 Apr 2013 21:31:19 GMT

And same here: no VM, memory at 90% capacity, and often CPU usage peaking towards 100%. Stopping services does not make a difference.
RJ

Oh, and I noticed that stopping the ArcGIS Server and ArcGIS Web Applications Manager brings down memory use from 9Gb out of 10 to 1.6-ish. Turning them back on doesn't bring them right back up, but CPU is close to 100% and memory is creeping up pretty fast...
RJ

Re: 10.1 Memory Issues

ElizabethNolan — Thu, 18 Apr 2013 12:58:42 GMT

We have been testing performance on our new 10.1 server, and trying to tune our new server(s) to handle our volume, both optimally and efficiently. Through testing we have come across an odd occurrence. When a geocode service is being accessed on the ArcGIS server, the number of ArcSOC.exes in use double. As a side effect, the memory use spikes and reaches maximum capacity. I talked to tech support and they told me that ArcSOC.exe was acting as designed. I know we can throw more memory on the server, but that is not the point. We can also adjust the runtime memory limit and the maximum instance capacity, and these changes might make a difference. But for a transactional environment changing memory limitation or capacities cause slower processing times, and bottle necks, and even with these changes the doubling of ArcSOC.exe would still occur.
In my opinion, this is poor design, and seems to be left over coding from earlier versions of ArcGIS, when DCOM was being utilized. I see no reason, nor have I obtained a valid reason, as to why ArcSOC.exe needs to double in use when a geocoding service is being accessed. Something is off in the invocation of the services, in my opinion.
An example, of this happening is, the calling process is sending 5 requests, and there are 5 instances in the management console in use. The monitor of the OS shows that there are 10 ArcSOC.exe running for the 5 instances. We are using an isolation plan so nothing else is running on the box. We have it in lockdown as we are testing the performance and tweaking.

Have any other developers noticed large memory usages, or that ArcSOC.exe doubles in size when geoprocessing?

Maybe if enough developers complain about this fact, then a solution will be created.

Re: 10.1 Memory Issues

KevinHibma — Thu, 18 Apr 2013 15:08:16 GMT

Just to be clear - you originally mention geocode service, then at the end you say geoprocessing service.
I've never seen an instance that a Geoprocessing service has doubled its memory footprint. It will take on more RAM resources if you compare the footprint of just started vs. used a few times. Depending what it does it may take on different amount of resources if the execution is somewhat dynamic (that is, not all executions follow the same code path).

As for the geocode service - I'm not an expert in this area, but I do know this can be a more RAM intensive service. Depending on the locator settings, the size of streets/locator, you could use up a lot of memory. It follows the same pattern of any other service. At startup it has a memory footprint and this footprint will grow some during execution. Again, I can't say for your particular situation how much it will grow, just from personal experiences I do know it can use more RAM (compared to say a Geoprocessing service).

Re: 10.1 Memory Issues

ElizabethNolan — Thu, 18 Apr 2013 15:51:26 GMT

I mistakenly put, geoprocessing, my bad.

What I am referring to is a geocode service. And it is memory intensive, but no one has been able to explain why there are double exes when it's running. Of course it would make sense that it would be more memory intensive since the runtime memory limit can be set up to 2 gig.

Re: 10.1 Memory Issues

eddiequinlan — Fri, 17 May 2013 20:27:27 GMT

I too am experiencing the same memory issues, which everyone has already stated above. Unless anyone has found an answer, our memory will continue to be gobbled up by the ArcSoc processes and the two javaw.exe processes. Somewhere on the net, i saw a post which basically said the javaw.exe processes should only be consuming a 1mb or less. On our server it's near the 1GB level. Sure would like to see this issue resolved.

Eddie

Re: 10.1 Memory Issues

SteveTimbrook — Mon, 20 May 2013 20:52:59 GMT

We are also having an issue with the memory growing on certain services. In our case we are seeing this issue with services that are referencing ECW images. Has anyone else seen this?
Steve

Re: 10.1 Memory Issues

ToraJohnson — Thu, 13 Jun 2013 13:54:32 GMT

Help! We're experiencing the same memory issue with zillions of SOC instances and huge Java processes. It's rendered our server unusable. It looks like no one on this thread has found a solution, but I'm posting just in case.

Thanks!

Re: 10.1 Memory Issues

KeG — Thu, 13 Jun 2013 14:46:45 GMT

don't have an answer, just suggestions. have you tried Kevin's suggestion:

Just a tip, which may or may not be useful in this situation:
If you goto Task Manager, and under VIEW > Select Columns.
Turn on "Command Line".
In the processes tab you can then see in for each ArcSOC.exe process which service it is running.
The name will be found at DService = "...."
This might be useful in helping to identify which services require more memory than others and planning deployment.

Also, have you tried ramping up the services one at a time to see how that affects things.
and can you align 'heavy' map documents with 'server intense' services.
i think there is also a way to set zero instances for a service, despite the fact that they are all pooled. not sure what zero instances means for a pooled service - i guess maybe it doesn't have an instance until the first call?