Map services frequently timing out

[IreneEgbulefu1]

We are running Enterprise ArcGIS 10.9.1 on Windows Server 2016 now upgraded to Windows Server 2022

We suddenly started noticing that the Memory consumption on the ArcGIS server is spiking upto 99% and on checking we noticed that the ArcSOC is generating in tons up to 224 ArcSOCs when we checked the counts.

We don't know what could be causing the spikes, because we haven't done anything different in the environment.

All our servers are on Virtual Machine

What we have done so far in the process of troubleshooting

1. We had to upgrade the memory on the Web adaptor based on one of the recommendations we found here - --- problem persisted

2. Installed all the Esri Patches on all the servers - Including the Portal for ArcGIS, Datastore and ArcGIS Server

3. Upgraded and updated the Windows Server operating system from Windows 2016 to Windows 2022

4. Our Server is a VM machine and the memory is 32GB, according to our ITS, it cannot take any more memory as that is the max for it

4. Contacted Esri Analyst support for help  - They are yet to find solution

5. We have also changed some of our Map services that are infrequently used from Dedicated to Shared services, still that did not provide a lasting solution

So, I am here to seek for help, if anyone has had such experience

[Town_of_Fuquay-Varina_GIS]

I'm curious in your testing, when you restart the ArcGIS Server windows service, do the 200+ ArcSOCs immediately spool up, or does it happen over time? Do you see anything in the ArcGIS Server logs viewable from ArcGIS Server Manager or Admin menu?

[IreneEgbulefu1]

when you restart the ArcGIS Server windows service, do the 200+ ArcSOCs immediately spool up, or does it happen over time? The ArcSOCs immediately spool up

Do you see anything in the ArcGIS Server logs viewable from ArcGIS Server Manager or Admin menu? Yes

We also use the Task Find command to get the counts the ArcSOCs from the command prompt

[David_McRitchie]

Hey Irene,

Just to add, with that volume of ArcSOCs we could potentially be hitting limitations on the number of Windows Processes that can be open at any one time.

Since services are on dedicated I would recommend going through and changing the min instances down to 0, and 1 max. This should cut down the number of active ArcSOCs considerably. We might still get issues if users begin spreading requests over all services but it should prevent ArcSOCs from draining memory on a machine boot.

I will warn that you should apply this advice subjectively. Having 0 min instances will mean services have to spin up an ArcSOC once requests come in. This will mean services will be slow with an approximate 30s spin up time. If a service needs to be readily available then we should keep the min instance setting to 1 or above.

Hope that helps,

David