nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Map services frequently timing out

IreneEgbulefu1

We are running Enterprise ArcGIS 10.9.1 on Windows Server 2016 now upgraded to Windows Server 2022

We suddenly started noticing that the Memory consumption on the ArcGIS server is spiking upto 99% and on checking we noticed that the ArcSOC is generating in tons up to 224 ArcSOCs when we checked the counts.

We don't know what could be causing the spikes, because we haven't done anything different in the environment.

All our servers are on Virtual Machine

What we have done so far in the process of troubleshooting

1. We had to upgrade the memory on the Web adaptor based on one of the recommendations we found here - --- problem persisted

2. Installed all the Esri Patches on all the servers - Including the Portal for ArcGIS, Datastore and ArcGIS Server

3. Upgraded and updated the Windows Server operating system from Windows 2016 to Windows 2022

4. Our Server is a VM machine and the memory is 32GB, according to our ITS, it cannot take any more memory as that is the max for it

4. Contacted Esri Analyst support for help - They are yet to find solution

5. We have also changed some of our Map services that are infrequently used from Dedicated to Shared services, still that did not provide a lasting solution

So, I am here to seek for help, if anyone has had such experience

image (1).png

image.png

ArcSOCS.png

Finde mehr Beiträge mit dem Tag

Accepted answers

All comments

David_McRitchie

Hey Irene,

Just to add, with that volume of ArcSOCs we could potentially be hitting limitations on the number of Windows Processes that can be open at any one time.

Since services are on dedicated I would recommend going through and changing the min instances down to 0, and 1 max. This should cut down the number of active ArcSOCs considerably. We might still get issues if users begin spreading requests over all services but it should prevent ArcSOCs from draining memory on a machine boot.

I will warn that you should apply this advice subjectively. Having 0 min instances will mean services have to spin up an ArcSOC once requests come in. This will mean services will be slow with an approximate 30s spin up time. If a service needs to be readily available then we should keep the min instance setting to 1 or above.

Hope that helps,

David

IreneEgbulefu1

when you restart the ArcGIS Server windows service, do the 200+ ArcSOCs immediately spool up, or does it happen over time? The ArcSOCs immediately spool up

Do you see anything in the ArcGIS Server logs viewable from ArcGIS Server Manager or Admin menu? Yes

We also use the Task Find command to get the counts the ArcSOCs from the command prompt

Town_of_Fuquay-Varina_GIS

I'm curious in your testing, when you restart the ArcGIS Server windows service, do the 200+ ArcSOCs immediately spool up, or does it happen over time? Do you see anything in the ArcGIS Server logs viewable from ArcGIS Server Manager or Admin menu?

IreneEgbulefu1

Hi Chris,

In response to your questions, pls see below

Since your default instance type for new services is "dedicated": How many services do you have on the server. - We have a good number of services- about 100
What are the default min./max. number of instances for a services ? default mini/max of instances 0/2, very few have 1/3 though not up to 10 services
Does the Windows Service run under a Domian User, a Windows Managed User or under "local System" Yes - Domain user

Usualy ArGIS Server only spins up a SOC if there is a request to a service. So either there are a lot of requests generated somewhere or maybe something prevents the server from shutting down the SOCs. that's what we are yet to figure out

meanwhile we have resorted to setting up additional ArcGIS Server site, hoping that would resolve the issue as recommended by Esri.

About to set up a test environment for that

GIS-Chris

Hi @IreneEgbulefu1
A few questions :

Since your default instance type for new services is "dedicated": How many services do you have on the server.
What are the default min./max. number of instances for a services ?
Does the Windows Service run under a Domian User, a Windows Managed User or under "local System"

Usualy ArGIS Server only spins up a SOC if there is a request to a service. So either there are a lot of request generated somewhere or maybe something prevents the server from shuting down the SOCs.

IreneEgbulefu1

Yes, I do and here is a screenshot of the page

arcGIS Server Pooling.png

CalvinHarmin

Do you have access to the ArcGIS Server manager menus? I'm curious what your settings are for Site > Settings > Pooling

IreneEgbulefu1

I have never had to do that, Please how is it done?

CodyPatterson

Hey @IreneEgbulefu1

I see I see, the server definitely is the culprit here, have you considered a possible rollback to pre-update to see if that would resolve the issue?

Cody

IreneEgbulefu1

Thanks Cody,

Yes I have access to the ArcGIS server.

We started noticing the issue on August 20th after ITS did their routine Windows update.

I will post a screenshot of the Taks manager now with the memory column sorted

Task1.PNG

Task2.PNG

CodyPatterson

Hey @IreneEgbulefu1

When you view the task manager, if you can sort by clicking the Memory column, that would let you check out what's consuming all the RAM, is it a single item or is it all ArcSOCs that are spread across the system? Do you have access to ArcGIS Server Manager, if so, I would check the instances on your services, typically an ArcSOC is associated with a service or a dedicated instance.

I'm interested to hear why your ITS mentions that the RAM being at 32GB is the supposed max for the system, my system regularly goes past 32GB under heavy loads.

When did this start happening? Was it a slow occurrence of it going more and more through the day, or did this happen all at once. Do you happen to have ArcGIS Monitor, we may be able to see the usage history and find what could be using all of the resources!

Just some things to think about, hopefully we can get it sorted!

Cody

Nimm an der Unterhaltung teil

Angemeldete Mitglieder können Beiträge verfassen, Updates folgen und mehr. Neu hier? Registriere ein kostenloses Konto.

Schnellzugriffe

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.