ArcGIS Server 11.4 CPU Usage

[MatthewRyan2]

Hey All, 

I have a problem with my brand new ArcGIS Server 11.4 installation. (fresh Windows Server 2019 machine)

I've installed all my new APRX which were previously MXD. Then we've starting using it as we did before and discovered that "normal usage" will occasionally max out the CPU and then the server will stay maxed out even after the users have logged off and gone home. The ArcGIS Server becomes unresponsive in all respects at this point.

Even when we manage the SoCs and move some into shared instances where it's appropriate, there is still a chance that this issue will occur.

On the topic of ArcGIS Server/Enterprise maxing our CPU and crashing, it seems to be a common occurrence. I've just read these two threads on it:

• https://community.esri.com/t5/arcgis-enterprise-portal-questions/100-cpu-usage-portal-for-arcgis-ser...
• https://community.esri.com/t5/arcgis-enterprise-questions/just-upgraded-to-11-3-and-now-openjdk-plat...

I have two issues with the above behaviours of the server and explanations found in those threads:

1. This continues to occur after ALL USERS are off the system (ie. nobody's sending any requests to the server)
2. This behaviour wasn't occurring with the ArcGIS .MXD runtime (The aprx runtime is completely different code)

#1 is the main concern, this can't be "too many SoCs" because there's nobody sending request to the server.  This are actually "runaway processes" which ArcGIS server has lost track of and they just keep churning away.

The description of "Runaway processes" as a cause to an outage would normally be considered as a bug in  software systems. I'm wondering why there hasn't been more of a spotlight here.

My questions for this are:

• Has anyone else gone through the process with ESRI discussing these points?
• Do we know if there is any consideration of it becoming a bug?
• Are there any patches out that fix this problem?

NOTE: I've tested this with 10.9 and 11.3 as well. On Windows 2016 Servers and 2012 Servers. Upgrade in place and brand new machines.

[berniejconnors]

We had a similar problem with an upgrade.  We went from two ArcGIS servers running 10.2.2 to three ArcGIS Servers running 10.7.1 on new VMs.  It was great to have the new system on new VMs so we could do comparisons between the old and new system.  You have to figure out where the bottleneck is that is slower in the new system compared to the old system.  You can't just throw up your hands and complain that it is slower.  It took us a lot of time and benchmarking but we figured out it was our file server.  The location for the config-store required OpLocks to be OFF.  The location for our FGDB required OpLocks to be ON.

The problem is likely different in your situation so you are going have to dig in and find the bottleneck.  We were lucky - we had a networking genius on our IT team to help with the diagnosis.  But it still took time.

Bernie.

[VerstraetenTim]

Hi, in the environment I am working on, we are experiencing the same issue in 11.1. It is a really a nuissance, because even though the server is pretty well spec'd and it is really impacting the production environment. 

Here's a list a things we did, all to no or little avail:

• increase RAM
• increase memory
• added sockets
• decrease the number of minimum instances for less used services
• moved less used services to a shared instance
• increased the number of recycles for the service instances
• limit idle runtime of service instances
• increase recycling of the IIS apppool

I am now in contact with our local ESRI representative to look at the issue. I hope it gets resolved soon, because I am burning through goodwill of our business here 🙂

TimV.

[RiceAdam]

@JoshuaBixby @bernsk If it helps at all... we experienced memory leaks after upgrading to 11.3 which was only resolved after the Security Patch 2 B. Our Available Memory also dropped from 10gb to 6gb. Interestingly, when we did have the memory leak our RAM would start higher at ~8gb before dropping by ~500mb-1gb/day. With the Security Patch we start on 6gb...

[JoshuaBixby]

Very interesting results, thanks for sharing.  I will have to revisit the issues fixed in that security patch to see what may have been the defect causing the memory issues.

[RiceAdam]

I cannot find any mention of a known Memory Leak when upgrading versions (or introduced in Security Patch 1); nor a fix in the Security Patch 2 B documentation, but it was/is occurring across multiple envs. I've raised this with ESRI.

The problem from my end is we're on AWS; and Security Patch 2 did not improve our confidence with the whole "we broke your Portal Index on S3".