100% CPU Usage - Portal for ArcGIS Server 10.7.1/10.8.1

[ScottMinter]

Installed portal for ArcGIS server 10.7.1 a month ago. No issues until noon today. At that point the CPU usage shot to 100% and has stayed that way.

ticket with Esri and worked on the issue all day. Still 100% cpu. Did a Repair on the software installation, rebuilt the portal index, updated database statistics...nothing has worked.

If the Portal service is stopped (windows) the cpu usage stops.

Nothing in the error logs seems out of place.

Anyone ever see anything like this or have ideas where to look? The highest percentage cpu usage seems to be java but it rotates and is not one process. 

[ScottMinter]

Updated Portal for ArcGIS 10.7.1 with patch "Portal for ArcGIS 10.7.1 Enterprise Sites Patch" and the cpu is no longer pegged at 100%.

[ScottMinter]

Well, it didn't work. Five days later the cpu pegged 100% again on a Saturday. We only work M-F so no users were on the system. Upgraded to 10.8.1. CPU back to 5%....for a few days, then 100% cpu again. Loaded all patches for 10.8.1 cpu back to normal....for a few days. Today, 100% again.

[ScottMinter · Accepted solution]

According to Esri support we need to install Datastore, a separate program. We have been a single machine since version 10.1 of ArcGIS Server. Only recently added portal. Then we were told we had to federate. Now we have to install datastore. All this because we just wanted to use Collector. So our simple server of just ArcGIS server 10.7.1 and web adaptor is now (all 10.8.1) ArcGIS Server, Portal for ArcGIS Server, Datastore, Web Adaptor. Just found a white paper that I think I'll read before making any other changes: https://www.esri.com/content/dam/esrisites/en-us/media/whitepaper/migrating-arcgis-server-to-arcgis-...

Following the steps in the white paper got us to a stable state and the issue has not returned.

[MaryGraceMcClellan]

Thanks for keeping this post updated, @ScottMinter!  Did upgrading to a 10.8.1 Base Enterprise Deployment permanently resolve the issue? 

[ScottMinter]

I read this white paper: https://www.esri.com/content/dam/esrisites/en-us/media/whitepaper/migrating-arcgis-server-to-arcgis-...

Following the steps in the white paper got us to a stable state and the issue has not returned.

[MarkGiesbrecht]

@ScottMinter  I'm curious to hear what the rogue processes are that are pinning the CPU.  We are suffering from this in our test environment (Enterprise 10.8.1, fully patched), where WMI Provider Host can suck up 40% of the CPU.  We've been troubleshooting for the last nine months, but to no avail.  Esri support suggests we have too many map and feature services running for the number of cores we have on the server.

[ScottMinter]

The issue was resolved for us by reading a white paper and installing more software to get us in the correct supported state (with Datastore, et. al.). Without following the steps in the white paper Esri was unsure what was going on. I don't remember what processes were running at 100%. I can tell you it was horrible and could take quite a long time just to reboot so we could troubleshoot.

[MaryGraceMcClellan]

@MarkGiesbrecht Did they give you an approved service:core number? I'm curious if that's our issue, but I haven't been able to find a number. 

[MarkGiesbrecht]

@MaryGraceMcClellan Here is what they sent back:

So I checked with the team. They confirmed that despite the RAM and hard disk being good, that the number of arcoc processes that you are running per the number of cores that you have is too many. This could be one of the reasons that the CPU is heavily loaded.

http://wiki.gis.com/wiki/index.php/Server_Software_Performance#Service_instance

I have attached a document that lists ways of managing this. 

You can try setting the less frequently used services to a minimum of 0 instances to see if this would reduce the load on your CPU. 

You can also try using shared instances if publishing from Pro.

https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/shared-instances-arcgis-s...

We have about 165 services on a 4 core server, and are beginning the slow process of publishing using ArcGIS Pro instead.  With some initial testing, the new Shared Instance configuration does reduce the server usage.