nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

ArcGIS Server 11.4 CPU Usage

MatthewRyan2

Hey All,

I have a problem with my brand new ArcGIS Server 11.4 installation. (fresh Windows Server 2019 machine)

I've installed all my new APRX which were previously MXD. Then we've starting using it as we did before and discovered that "normal usage" will occasionally max out the CPU and then the server will stay maxed out even after the users have logged off and gone home. The ArcGIS Server becomes unresponsive in all respects at this point.

Even when we manage the SoCs and move some into shared instances where it's appropriate, there is still a chance that this issue will occur.

On the topic of ArcGIS Server/Enterprise maxing our CPU and crashing, it seems to be a common occurrence. I've just read these two threads on it:

I have two issues with the above behaviours of the server and explanations found in those threads:

This continues to occur after ALL USERS are off the system (ie. nobody's sending any requests to the server)
This behaviour wasn't occurring with the ArcGIS .MXD runtime (The aprx runtime is completely different code)

#1 is the main concern, this can't be "too many SoCs" because there's nobody sending request to the server. This are actually "runaway processes" which ArcGIS server has lost track of and they just keep churning away.

The description of "Runaway processes" as a cause to an outage would normally be considered as a bug in software systems. I'm wondering why there hasn't been more of a spotlight here.

My questions for this are:

Has anyone else gone through the process with ESRI discussing these points?
Do we know if there is any consideration of it becoming a bug?
Are there any patches out that fix this problem?

NOTE: I've tested this with 10.9 and 11.3 as well. On Windows 2016 Servers and 2012 Servers. Upgrade in place and brand new machines.

Find more posts tagged with

Accepted answers

All comments

RiceAdam

I cannot find any mention of a known Memory Leak when upgrading versions (or introduced in Security Patch 1); nor a fix in the Security Patch 2 B documentation, but it was/is occurring across multiple envs. I've raised this with ESRI.

The problem from my end is we're on AWS; and Security Patch 2 did not improve our confidence with the whole "we broke your Portal Index on S3".

JoshuaBixby

Very interesting results, thanks for sharing. I will have to revisit the issues fixed in that security patch to see what may have been the defect causing the memory issues.

RiceAdam

@JoshuaBixby @bernsk If it helps at all... we experienced memory leaks after upgrading to 11.3 which was only resolved after the Security Patch 2 B. Our Available Memory also dropped from 10gb to 6gb. Interestingly, when we did have the memory leak our RAM would start higher at ~8gb before dropping by ~500mb-1gb/day. With the Security Patch we start on 6gb...

VerstraetenTim

Hi, in the environment I am working on, we are experiencing the same issue in 11.1. It is a really a nuissance, because even though the server is pretty well spec'd and it is really impacting the production environment.

Here's a list a things we did, all to no or little avail:

increase RAM
increase memory
added sockets
decrease the number of minimum instances for less used services
moved less used services to a shared instance
increased the number of recycles for the service instances
limit idle runtime of service instances
increase recycling of the IIS apppool

I am now in contact with our local ESRI representative to look at the issue. I hope it gets resolved soon, because I am burning through goodwill of our business here 🙂

TimV.

JoshuaBixby

Although having insufficient paging space can cause memory issues, so too can having too little physical RAM. The page file and physical RAM work together, and neither one can be a complete substitute for the other. (I am dismissing the old "I have so much RAM I disabled my page file" argument because that is a terrible idea for many reasons, especially in production server environments.) There is a limit to how much of a process address space can be paged, i.e., some portion of the process address space always has to remain in physical memory. Therefore, someone can have a 10 TB page file and still crash an application or system due to too little physical memory.

Joey_NL

Interesting to hear! Feedback we've had on a support ticket was that ArcGIS GIS Server doesn't make use of page space, and that matches what we've seen in production. The server will run itself into the ground rather than use any of the swap file. Interesting to see that the experience with ArcGIS Enterprise seems to be different.

berniejconnors

@JoshuaBixby , Is there any documentation or warnings from Esri that ArcGIS Server 11.x requires more RAM to run the same number of services as ArcGIS Server 10.x? This thread is the first time I have come across this information.

berniejconnors

Thanks Joshua. That is great info for the planning of our 11.5 upgrade.

JoshuaBixby

Memory configurations are very specific to machines and applications, so it is hard to give recommendations without knowing quite a bit more information. While allowing the OS to wholly manage the virtual memory settings is convenient, and maybe sufficient for some groups/organizations given their situations, I doubt you see it being used in many enterprise IT operations.

What I can say at this point is that ArcGIS Server 11.x uses more memory than ArcGIS Server 10.x given the exact same services and settings. Whether that increase requires more RAM or different virtual memory settings on a specific server depends on the amount of existing RAM and existing virtual memory settings.

I would encourage your IT folks to monitor various memory counters over time, simply pulling up Task Manager and taking a look a few times during the day is not sufficient. A spike in traffic can cause a spin-up of more ArcSOC.exe processes which can lead to a momentary spike in memory usage. If the spike is high enough, it can cause lasting instability in the ArcGIS Server framework processes. Memory spikes can also be brief enough that casually looking at Task Manager throughout the day won't necessarily catch them.

berniejconnors

@JoshuaBixby wrote:
The organization I used to work for initially had instability problems with ArcGIS Enterprise 11.x until we analyzed memory utilization of the ArcGIS Enterprise processes and subprocesses. It turns out the ArcGIS Pro runtime used in ArcGIS Enterprise 11.x results in ArcSOC.exe processes having about 50% increase in total memory footprint. I say total memory footprint because it wasn't more RAM being used but more virtual address space by the processes. It turns out we were hitting or nearly hitting the committed memory limit of the server, and the entire ArcGIS Enterprise framework would become unstable at that point. Once it became unstable, the only option to stabilize it was to stop the ArcGIS Server Windows service and restart it.
I suggest you monitor the Committed memory usage of the server, and not just RAM. If you are getting over 90% committed memory for any period of time, I would update the virtual memory (page file) settings on the server.

Just the other day I had one of our server admins tell me:

"With the latest versions of Windows Server we prefer to let the OS manage the size of the page file".

That was a general comment on Windows Server 2022 - not specific to ArcGIS Enterprise. I had asked the server admin to collect information on the paging file size from our 3 existing VMs running ArcGIS Server on older Windows Server 2016 VMs.

@JoshuaBixby is your recommendation to increase the page file size?

Bernie.

berniejconnors

We had a similar problem with an upgrade. We went from two ArcGIS servers running 10.2.2 to three ArcGIS Servers running 10.7.1 on new VMs. It was great to have the new system on new VMs so we could do comparisons between the old and new system. You have to figure out where the bottleneck is that is slower in the new system compared to the old system. You can't just throw up your hands and complain that it is slower. It took us a lot of time and benchmarking but we figured out it was our file server. The location for the config-store required OpLocks to be OFF. The location for our FGDB required OpLocks to be ON.

The problem is likely different in your situation so you are going have to dig in and find the bottleneck. We were lucky - we had a networking genius on our IT team to help with the diagnosis. But it still took time.

Bernie.

JoshuaBixby

The organization I used to work for initially had instability problems with ArcGIS Enterprise 11.x until we analyzed memory utilization of the ArcGIS Enterprise processes and subprocesses. It turns out the ArcGIS Pro runtime used in ArcGIS Enterprise 11.x results in ArcSOC.exe processes having about 50% increase in total memory footprint. I say total memory footprint because it wasn't more RAM being used but more virtual address space by the processes. It turns out we were hitting or nearly hitting the committed memory limit of the server, and the entire ArcGIS Enterprise framework would become unstable at that point. Once it became unstable, the only option to stabilize it was to stop the ArcGIS Server Windows service and restart it.

I suggest you monitor the Committed memory usage of the server, and not just RAM. If you are getting over 90% committed memory for any period of time, I would update the virtual memory (page file) settings on the server.

Joey_NL

Very interesting that you say this! This matches our experience with 11.2 as well.

We have a lot of services which we migrated from a 10.6 server / ArcMap runtime and now are facing incredibly poor performance on a 11.2 server.

We have been trying with ESRI since last year to progress this as a bug, and the answer was just - More RAM, More Cores.

I'd be incredibly interested in hearing if there is a bug or a patch for this. We've wasted so much time.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.