ArcSOC Processes with 11.5 Upgrade

[JohnCBowers]

Hello,

I just upgraded our Enterprise deployment from 11.3 to 11.5 last weekend. Everything validated fine over the weekend, but once people started working on Monday we got heavy CPU spikes on our hosting server, along with hosting store validation failures on our Portal page, the failure being 'failed to connect to registered datastore'.  The datastore also failed to validate within the server manager, but always successfully validates within the server admin page.

We have unregistered and re-registered the datastore and the problem still persists. 

I did notice that our SOC heap was set to 256MB, so I lowered that to 128MB and that seems to have cut down the hosting server CPU spikes, which in turn is showing successful validation across the board. But once the CPUs spike, the validation fails with the same datastore failure, but it appears everything still works within our deployment.

I do see in my hosting server machine, what looks like over 100 ArcSOC processes running within the task manager. I 'think' that is the cause of the CPU spikes in the hosting server. 

What can I do to cut down on these ArcSOC processes?

Thank you so much. 

[CodyPatterson]

Hey @JohnCBowers 

Depending on what services are set to a "dedicated" instance, that may be driving up the ArcSOC processes, I'd check the Server Manager to see if there are any services running on a dedicated pool and verify that they need to run on them, or if they can be moved to shared. Along with that, what do the resources look like on the Server machine? I had this same issue, and bumped the CPU cores up and it resolved it in the past.

Cody

[A_Wyn_Jones]

You should switch rarely used Map services to the shared instance pool. Most machines have 4 ArcSOC processes assigned to the shared instance pool - this means that 4 ArcSOC processes will be "driving" all Map services assigned to the shared instance pool.

https://enterprise.arcgis.com/en/server/latest/administer/windows/configure-service-instance-setting...

This article contains a lot of detail on the topic. Recommend you switch the publishing default to shared too.

You can use ArcGIS Server statistics to find out which services are regularly/rarely used which will identify candidates for switching the instance pools - inversley this will tell you which services need to be Dedicated instance pools

https://enterprise.arcgis.com/en/server/latest/administer/windows/about-server-statistics.htm

Keep in mind that with each software release, there's more product and therefore more to run! I have seen the issues you've described after upgrades which were resolved by increasing RAM and CPU 

[TravisSaladino]

A quick suggestion for the question, "What can I do to cut down on these ArcSOC processes?" You might want to look at moving many of the services to the shared instance pool. Here's a blog article about a python tool you could use
https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/move-services-to-shared-i... 
or you can do it manually too.