Map services frequently timing out

[IreneEgbulefu1]

We are running Enterprise ArcGIS 10.9.1 on Windows Server 2016 now upgraded to Windows Server 2022

We suddenly started noticing that the Memory consumption on the ArcGIS server is spiking upto 99% and on checking we noticed that the ArcSOC is generating in tons up to 224 ArcSOCs when we checked the counts.

We don't know what could be causing the spikes, because we haven't done anything different in the environment.

All our servers are on Virtual Machine

What we have done so far in the process of troubleshooting

1. We had to upgrade the memory on the Web adaptor based on one of the recommendations we found here - --- problem persisted

2. Installed all the Esri Patches on all the servers - Including the Portal for ArcGIS, Datastore and ArcGIS Server

3. Upgraded and updated the Windows Server operating system from Windows 2016 to Windows 2022

4. Our Server is a VM machine and the memory is 32GB, according to our ITS, it cannot take any more memory as that is the max for it

4. Contacted Esri Analyst support for help  - They are yet to find solution

5. We have also changed some of our Map services that are infrequently used from Dedicated to Shared services, still that did not provide a lasting solution

So, I am here to seek for help, if anyone has had such experience

[CodyPatterson]

Hey @IreneEgbulefu1 

When you view the task manager, if you can sort by clicking the Memory column, that would let you check out what's consuming all the RAM, is it a single item or is it all ArcSOCs that are spread across the system? Do you have access to ArcGIS Server Manager, if so, I would check the instances on your services, typically an ArcSOC is associated with a service or a dedicated instance.

I'm interested to hear why your ITS mentions that the RAM being at 32GB is the supposed max for the system, my system regularly goes past 32GB under heavy loads.

When did this start happening? Was it a slow occurrence of it going more and more through the day, or did this happen all at once. Do you happen to have ArcGIS Monitor, we may be able to see the usage history and find what could be using all of the resources!

Just some things to think about, hopefully we can get it sorted!

Cody

[IreneEgbulefu1]

Thanks Cody,

Yes I have access to the ArcGIS server.

We started noticing the issue on August 20th after ITS did their routine Windows update.

I will post a screenshot of the Taks manager now with the memory column sorted

[CodyPatterson]

Hey @IreneEgbulefu1 

I see I see, the server definitely is the culprit here, have you considered a possible rollback to pre-update to see if that would resolve the issue?

Cody

[IreneEgbulefu1]

I have never had to do that, Please how is it done?

[CodyPatterson]

Hey @IreneEgbulefu1 

This would more than likely require the assistance of your ITS, I believe that means your IT department or similar, they would need to go into the server settings, and roll back the most recent update to your server, which would require a restart in most cases. I am interested in your response to CalvinHarmin though as well!

Cody

[CalvinHarmin]

Do you have access to the ArcGIS Server manager menus? I'm curious what your settings are for Site > Settings > Pooling

[IreneEgbulefu1]

Yes, I do and here is a screenshot of the page

[GIS-Chris]

Hi @IreneEgbulefu1 
A few questions :

1. Since your default instance type for new services is "dedicated": How many services do you have on the server.
2. What are the default min./max. number of instances for a services ?
3. Does the Windows Service run under a Domian User, a Windows Managed User or under "local System"

Usualy ArGIS Server only spins up a SOC if there is a request to a service. So either there are a lot of request generated somewhere or maybe something prevents the server from shuting down the SOCs. 

[IreneEgbulefu1]

Hi Chris,

In response to your questions, pls see below

1. Since your default instance type for new services is "dedicated": How many services do you have on the server. - We have a good number of services- about 100
2. What are the default min./max. number of instances for a services ? default  mini/max of instances  0/2, very few have 1/3 though not up to 10 services
3. Does the Windows Service run under a Domian User, a Windows Managed User or under "local System" Yes - Domain user 

Usualy ArGIS Server only spins up a SOC if there is a request to a service. So either there are a lot of requests generated somewhere or maybe something prevents the server from shutting down the SOCs.  that's what we are yet to figure out

meanwhile we have resorted to setting up additional ArcGIS Server site, hoping that would resolve the issue as recommended by Esri.

About to set up a test environment for that