nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

10.1 Memory Issues

JustinGreco

I have installed 10.1 onto a new server and have been experiencing issues with memory usage. Our current 10 installation of ArcGIS Server utilizes about 12 MB of memory, while 10.1 is utilizing about twice the amount of memory with less services running than on our 10 server. Also the javaw.exe process grows to over 2,000,000 K. Has anyone else had issues like this with 10.1? Tech support did not seem aware of this issue.

Vind meer berichten getagd met

Accepted answers

All comments

KevinGooss

i have two javaw.exe processes - one is at 191 and the other is at 278,000K.
I also have 14 ArcSOC.exe processes at the moment, each of those is about 100,000K average.
This is a new box and if we add map services at the rate we normally do we are quickly going to be out of memory.

KevinHibma

I'm not 100% sure of this, but I think one of the javaw.exe is for logging.
Do you have logging set to a high level? If you set it back to a low level (like SEVERE), and re-start ArcGIS Server, it shouldnt be as active.

JustinGreco

Kevin,

I have the log level set at warning and just switched it to severe, but the javaw is still using over 1,000,000 KB. The other issue is the amount of memory ArcGIS Server is using at 10.1, almost twice was much as 10. Is that to be expected at 10.1? I also notice that when I stop the ArcGIS Server service, the memory usage does not drop at all.

KevinHibma

If you've stopped and re-started the ArcGISServer service in the Services panel, and memory didnt change, then the process sounds either hung or orphaned.
When you restart they should all disappear and then come back fresh.

After re-starting my server I have:

2 javaw.exe
1 arcgisserver.exe
X # of ArcSOC.exe

I'd expect you to have something similar.

I cant give you a yes/no answer on how much memory should be used. I don't concentrate on the core architecture and wouldnt want to give you a wrong answer.
Just my impression though, 1 gig (1,000,000KB) sounds high for the javaw.exe process, but without knowing exactly what your server is doing, # of requests, # of services, etc - its hard to say if thats a problem. The biggest change which I'm sure you're aware of is 10.1 being 64bit, so each process can potentially consume more memory than 10.0 -32bit server.

Just a tip, which may or may not be useful in this situation:
If you goto Task Manager, and under VIEW > Select Columns.
Turn on "Command Line".
In the processes tab you can then see in for each ArcSOC.exe process which service it is running.
The name will be found at DService = "...."
This might be useful in helping to identify which services require more memory than others and planning deployment.

JustinGreco

Turns out it wasn't an ArcGIS Server issue, but a setting that needed to be changed in our VM software after increasing the amount of RAM.

ChrisOrpurt

Turns out it wasn't an ArcGIS Server issue, but a setting that needed to be changed in our VM software after increasing the amount of RAM.

Can you post what that setting was?

Jay_Gregory

I'm having similar memory problems with a new install of Server 10.1 - any idea what VM setting you changed?

GeraldLee__GISP

Count me in with this same issue--my 10.1 memory usage is about 2x as same 10.0 services, stopping services doesnt seem to relieve CPU, etc..i also am using VMware..on windows server 2008 R2. My IT said i must be hallucinating...glad to know its not just me.. 🙂

EricKriener

I'm having similar memory problems with a new install of Server 10.1 - any idea what VM setting you changed?

I'm going to wake this thread up.

Anybody know what the VM setting to change is?

AndrewWharton

I just encountered the same issue with one instance of javaw.exe using over 3,200,000kb. Two other instances were in that 200,000kb range. We are using 10.1 but aren't using VM software, so it looks like an issue or setting that affects more than just VM.

RobertWeise

And same here: no VM, memory at 90% capacity, and often CPU usage peaking towards 100%. Stopping services does not make a difference.
RJ

RobertWeise

And same here: no VM, memory at 90% capacity, and often CPU usage peaking towards 100%. Stopping services does not make a difference.
RJ

Oh, and I noticed that stopping the ArcGIS Server and ArcGIS Web Applications Manager brings down memory use from 9Gb out of 10 to 1.6-ish. Turning them back on doesn't bring them right back up, but CPU is close to 100% and memory is creeping up pretty fast...
RJ

ElizabethNolan

We have been testing performance on our new 10.1 server, and trying to tune our new server(s) to handle our volume, both optimally and efficiently. Through testing we have come across an odd occurrence. When a geocode service is being accessed on the ArcGIS server, the number of ArcSOC.exes in use double. As a side effect, the memory use spikes and reaches maximum capacity. I talked to tech support and they told me that ArcSOC.exe was acting as designed. I know we can throw more memory on the server, but that is not the point. We can also adjust the runtime memory limit and the maximum instance capacity, and these changes might make a difference. But for a transactional environment changing memory limitation or capacities cause slower processing times, and bottle necks, and even with these changes the doubling of ArcSOC.exe would still occur.
In my opinion, this is poor design, and seems to be left over coding from earlier versions of ArcGIS, when DCOM was being utilized. I see no reason, nor have I obtained a valid reason, as to why ArcSOC.exe needs to double in use when a geocoding service is being accessed. Something is off in the invocation of the services, in my opinion.
An example, of this happening is, the calling process is sending 5 requests, and there are 5 instances in the management console in use. The monitor of the OS shows that there are 10 ArcSOC.exe running for the 5 instances. We are using an isolation plan so nothing else is running on the box. We have it in lockdown as we are testing the performance and tweaking.

Have any other developers noticed large memory usages, or that ArcSOC.exe doubles in size when geoprocessing?

Maybe if enough developers complain about this fact, then a solution will be created.

KevinHibma

Just to be clear - you originally mention geocode service, then at the end you say geoprocessing service.
I've never seen an instance that a Geoprocessing service has doubled its memory footprint. It will take on more RAM resources if you compare the footprint of just started vs. used a few times. Depending what it does it may take on different amount of resources if the execution is somewhat dynamic (that is, not all executions follow the same code path).

As for the geocode service - I'm not an expert in this area, but I do know this can be a more RAM intensive service. Depending on the locator settings, the size of streets/locator, you could use up a lot of memory. It follows the same pattern of any other service. At startup it has a memory footprint and this footprint will grow some during execution. Again, I can't say for your particular situation how much it will grow, just from personal experiences I do know it can use more RAM (compared to say a Geoprocessing service).

ElizabethNolan

I mistakenly put, geoprocessing, my bad.

What I am referring to is a geocode service. And it is memory intensive, but no one has been able to explain why there are double exes when it's running. Of course it would make sense that it would be more memory intensive since the runtime memory limit can be set up to 2 gig.

eddiequinlan

I too am experiencing the same memory issues, which everyone has already stated above. Unless anyone has found an answer, our memory will continue to be gobbled up by the ArcSoc processes and the two javaw.exe processes. Somewhere on the net, i saw a post which basically said the javaw.exe processes should only be consuming a 1mb or less. On our server it's near the 1GB level. Sure would like to see this issue resolved.

Eddie

SteveTimbrook

We are also having an issue with the memory growing on certain services. In our case we are seeing this issue with services that are referencing ECW images. Has anyone else seen this?
Steve

UMMGIS

Help! We're experiencing the same memory issue with zillions of SOC instances and huge Java processes. It's rendered our server unusable. It looks like no one on this thread has found a solution, but I'm posting just in case.

Thanks!

KeG

don't have an answer, just suggestions. have you tried Kevin's suggestion:

Just a tip, which may or may not be useful in this situation:
If you goto Task Manager, and under VIEW > Select Columns.
Turn on "Command Line".
In the processes tab you can then see in for each ArcSOC.exe process which service it is running.
The name will be found at DService = "...."
This might be useful in helping to identify which services require more memory than others and planning deployment.

Also, have you tried ramping up the services one at a time to see how that affects things.
and can you align 'heavy' map documents with 'server intense' services.
i think there is also a way to set zero instances for a service, despite the fact that they are all pooled. not sure what zero instances means for a pooled service - i guess maybe it doesn't have an instance until the first call?

DominicJames

We are finding that after caching a service, that CPU is 100% - the culprit appears to be .javaw. Restating the service appears to (eventually) solve the problem. This is obviously a bug.

I would be curious to know what settings were changed in the VM environment that solved this problem so we can test it.

ChrisCyphers_0-1760370119713.png

ChrisCyphers_1-1760370158368.png

ChrisCyphers_2-1760372851755.png

ArtUllman

I'm going to wake this thread up.

Anybody know what the VM setting to change is?

We had a similar problem. We originally started out with only 4GB of RAM on a VM, which was not enough. We requested more memory. Well, they allocated more virtual memory to the VM, but kept the same physical memory settings (without tell us this). As a result, the OS thinks it has more memory, but there is a balloon driver in VM that actually steals all of the fake memory so the applications do not get to use any of the memory (which doesn't exist anyway). We kept seeing memory utilization go up every time we requested more memory, because the balloon driver kept consuming the additional virtual memory. We thought it was ArcGIS, but was really the VM Settings. You can see the "private" balloon memory using a tool from sysinternals called RAMMAP.

DarrenMackiewicz

No help here Esri????

Did the original person who posted about changing the settings in the VM ever reply with an answer?
I'm currently experiencing this with a 10.1 ArcGIS Server implementation, so this IS still an issue.

BryanEvans

Warning: long post!

I wanted to chime in since we've done quite a bit of testing and have noticed a few things that may help some people on this thread.

Oh, and I noticed that stopping the ArcGIS Server and ArcGIS Web Applications Manager brings down memory use from 9Gb out of 10 to 1.6-ish. Turning them back on doesn't bring them right back up, but CPU is close to 100% and memory is creeping up pretty fast...
RJ

The reason this happens is because stopping/starting the ArcGIS Server windows service essentially restarts everything. So the memory drops substantially in a short amount of time because it drops all the ArcSOC.exe's. When the service starts, the CPU jumps to nearly 100% for a bit because it's recreating all those ArcSOC.exe (which correlate to service instances). I've noticed that depending on the number of services you have and minimum instances you have set, this could take some time. To make matters worse - if for whatever reason you have to restart the service mid-day while people are still trying to access it, this process could take longer because it could potentially bump up the number of instances if you have the minimum instances set low.

Help! We're experiencing the same memory issue with zillions of SOC instances and huge Java processes. It's rendered our server unusable. It looks like no one on this thread has found a solution, but I'm posting just in case.

Thanks!

I don't have a direct answer to fix your situation but it provides a good segue into listing what we've learned about ArcGIS Server 10.1. Hopefully it helps you understand the issues you having and how to possibly tackle them.

What we've learned (and I apologize if this somewhat common knowledge):

ArcGIS Server 10.1 is a complete architecture change/potential rewrite (at least I'm pretty sure). Server 10 running on Windows servers was written in .NET which explains the ability to use local server groups in security, connecting over port 80 without use of the Web Adaptor (because it was an IIS application), DCOM, etc. Server 10.1 looks to be based off Apache Geronimo (a Java based open source application server). That's why you see javaw.exe's (Java worker processes), it has the cluster ability, you need the Web Adaptor (running in IIS) to get Windows Authentication and traffic over port 80, etc.
The two javaw.exe's you see are Java worker processes like I mentioned. From all my installs it looks like one is dedicated to logging. I don't know the specific use of the other but it seems to be heavily used for some people. I'd encourage people to add the "Command Line" column in Task Manager like khibma mentioned. You should be able to tell from that column if it's the logging process or the other one.
I haven't done an apples to apples testing of Server 10 to 10.1, but I tend to agree that 10.1 does seem to use more overall memory. I don't know if it's purely the 64-bit aspect or other factors, but I will mention that we've personally noticed that Server 10.1 will increase service instances with nothing more than 1 or 2 requests (and increasing instances means creating a new ArcSOC.exe process which eats up more memory). How we found this was that we published a new service on a server no one was using but me. It was set to min instances of 2 and max of 4. When the service started, it created the 2 instances (I was monitoring via Server Manager). I made one map request to the service, watched the "Instances in use" go to 1 then back to 0, then watched as the min instances increased to 3 even though 2 were already running. I did this again a couple times and in a few minutes it was up to the max of 4 instances. I never caused the instances in use to go over 1, yet the number of instances rose. Maybe this was an odd case, but the moral is watch what your number of running instances are because that is one of the biggest factors in memory use.
To reduce the number of running instances you have a few options: restart the individual service (which will drop all running ArcSOC.exe's for that service on all machines in the site and start only the min instance on each machine), stop and start each machine in the site (this will drop all ArcSOC.exe's and the javaw.exe that isn't used for logging (note: the ArcGIS Server windows service and the javaw.exe for logging will still be running on the machine) and recreate them at the min instances per machine), or stop and start the ArcGIS Server windows service on each machine (this one will do the same as stopping the machine but also drop the ArcGIS Server.exe and javaw.exe used for logging).
To give an idea of memory usage for us, we have a Windows Server 2008 R2 VM using Hyper-V3 (says our server admin group) with 4 cores (Intel Xeon @ 3.07 GHz) and 24GB of memory. Right now I counted 86 ArcSOC.exe/running service instances and it's currently using 15.6 GB of memory (65% utilization). I'd say most of our services are medium-sized with some larger-sized services. We also have logging set to a higher output and the javaw.exe for logging is using 1.2 GB of RAM.
Kind of a general thing I've noticed is memory use is correlated with number of running instances (more instances mean more memory used). Service requests are what impact CPU usage. It can be a tricky thing to calculate out but you'll want an appropriate number of instances to satisfy requests so they don't queue up (causing CPU to get hammered while it renders map requests), but you don't want too many running instances (causing memory use to get hammered). You also have to take into account how long requests to your services take to return. If you have bloated services that take 20 seconds to complete and you get multiple requests, you'll quickly see CPU usage spike up and stay until they can complete. From things I've read, over 85% utilization constantly for either CPU or RAM severely degrades performance. Ideally memory utilization is relatively flat somewhere under 85% and CPU usage goes up and down but never 85%+ for an extended period of time. You'll see CPU usage go to 100% for 5-10 seconds at times but it shouldn't last minutes/hours.
If you have to use the Web Adaptor for multiple machine sites or for Windows Authentication (IIS only), that's another variable to watch. We've noticed flakiness in the IIS Web Adaptor. Sometimes it just gives out and bouncing IIS doesn't fix it. We restart IIS nightly but sometimes it still has issues. We haven't gotten with ESRI about anything specific about it yet but I've read other forum posts mentioning frustration with the Web Adaptor.

Sorry for the extremely long post but hopefully it helps some people understand what's going on with Server 10.1 and how to configure things to get better performance out of it.

bbieb

Just to add more memory information.
We have 2 identical VM machines, one running 10.0 SP2 and the other is running 10.2. Both machines have the same services running...which equates to about 100 ArcSoc.exes on each machine. On the 10.0 machine, the ArcSoc.exes average 77k and on the 10.2 machine, the ArcSOC.exes average 132k.

I've been talking to support about the increased memory usage and their response was that it is documented in the help. I was pointed to documentation for 10.0, http://help.arcgis.com/en/systemrequirements/index.html#/ArcGIS_Server_10_System_Requirements/015100000072000000/ and to the documentation of for 10.1, http://resources.arcgis.com/en/help/system-requirements/10.1/index.html#//015100000072000000 .

If you read very closely, the specs from the development environment changed from 2GB to 4GB. I guess from that you are suppose to infer about the increased memory usage between versions.

I was also given this bug number which was declined. NIM092919

This doesn't solve anyone's problems but just confirms what everyone is seeing. And you will see the same memory increase in 10.2.
brian

Scott_Prindle

As stated above, it is very helpful to enable the Command Line column in Task Manager. This will show more information related to each process. The ArcSOC.exe processes are all associated with your services, and this is revealed through the Command Line column. This can help put a finger on an individual service that may be consuming an extraordinary amount of memory.

The javaw.exe processes do a vast majority of the underlying ArcGIS Server work. The one that has "logging" in the Command Line is not necessarily solely for logging. It is likely the underlying java process that is managing and running your server.

Another place to look to ensure you aren't running out of system resources is the System Commit under the Performance tab in Task Manager. If this ratio is reaching 1 (ex. 15/16GB), then you are running out of virtual memory on the server. To get a better idea of the total memory consumption of each process, turn on the Commit Size column for the Processes tab. This will be a higher value than the Memory (Private Working Set). If you are seeing any performance issues and the System Commit is maxing out then you may need to provide more virtual memory for the machine. This can be done through Advanced System Properties>Performance Settings>Advanced>Change Virtual Memory. Ideally the virtual memory should be managed by the system, but it can also be increased to a more acceptable level.

Separate from the system properties themselves, service properties can also be modified to better suit the system they exist on. If a map service is rarely used, its minimum number of instances can be lowered to 0 or 1. Many services will be just fine with a maximum of 2-4 instances as well. If the server is a much larger deployment, then it would be prudent to use the Capacity Planning Tool:
http://wiki.gis.com/wiki/index.php/Capacity_Planning_Tool

Tuning and configuring services:
http://resources.arcgis.com/en/help/main/10.2/0154/015400000343000000.htm

MikeHargreaves

We have been having some similar issues. This morning, one of the javew.exe processes were using all of the CPU. We stopped the process, and now it seems to be fine. We've upped the RAM on the VM, and have had less issues like this. Before, we would see memory creep up, and then CPU max out, but it has not been a problem for a month or so until this morning. Any ideas what might have happened this time?

gareth_finney

We've got similar issues, and we're at 10.2 so extremely keen to find out from ESRI what we can do about this. Our logging javaw.exe process has reached 1.5+gig on a number of occasions and won't let up. We have about 20 map services hooked in, so not a massive load as far as SOCs are concerned, but that log process concerns me when I read that some users are getting 3+ gig tied up there!
We're about to upgrade to 16gig RAM, across 4 servers/nodes (quad core) in the 'site' and will be hitting that up with some decent load tests in the next day or two, so will report back some findings if that helps anyone!

First and foremost - is there a way of restricting the amount of memory used for the logging process?? it seems the other javaw.exe has a maximum memory set in the string, but the logging one doesn't - or is this not a good idea? Just looking for some feedback on this if possible.

cheers

sreeharikottoormadam

We are also facing the same issue in our QA( ArcGIS for Server 10.1 with SP1) and Prod servers(ArcGIS for Server 10.1 without SP1). Our IT Infra team informed they have received an alert �??Percentage of Committed Memory in Use is too high�?� for the servers and found multiple instances javaw.exe are running and consuming more memory of the server.

For QA we have single machine configuration and for Prod we have multiple machine configuration (each 8 GB memory) . Is anyone has found any solution to reduce the memory usage due to the javaw.exe process , please let us know.

Thanks ,
Sreehari

RandallWilliams

If you're getting reports that the percentage of committed memory is too high the solution is not to try to limit the memory consumed by the GIS Server, the solution is to increase the virtual memory available to the server. The commit charge percent is a relative measure that takes both physical and virtual memory into account.

sreeharikottoormadam

Hi Randall,

Many thanks for your advice. We have requested them to make the changes to increase the virtual memory.

Thanks & Regards,
Sreehari

FraserHand

We have something similar - did we get a response to this?

Thanks,

Fraser

SusanJones

Hi Eric

We have exactly the same issue after an upgrade from ArcGIS Server 10.2 to ArcGIS 10.3 2 weeks ago.

Same environment - VMWare and Windows Server 2008 R2.

Did you ever find a solution to this?

Is VMware an issue would Hyper Visor be a better choice of virtualisation?

Susan

AaronKreag

I am more a fan of Hyper V. Did you upgrade in place…using the same VMs? Can you detail your old environment and the new one, tell us what the symptoms are….

Doe mee aan het gesprek

Aangemelde leden kunnen berichten plaatsen, updates volgen en meer. Nieuw hier? Registreer een gratis account.

Snelle Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.