ArcGIS Server 10.8.1 windows service stops on its own

[BanchanaPandey]

We have following software installed individually on four machines

1 arcgis portal

2 federated non hosted arcgis server

3 federated hosted arcgis server

4 arcgisdatastore

The issue is with the federated arcgis server, second machine. We have noticed that on panning, zooming webmaps in a desktop browser, is causing the arcgis server to slow down and eventually services don't respond. the windows service for arcgis server is terminated, while we see the arcsoc processes are still around, almost all of them,

this machine has resources in terms of ram and cpu, page file space option is set to automatically grow.

On the IIS event viewer application log, we are seeing these errors in particular:

"Windows successfully diagnosed a low virtual memory condition. javaw.exe, arcsoc.exe and w3wp consumed x amount of bytes"

"ArcGIS Server terminated unexpectedly, it has done  this 2(tries) "

This is what the error in the dmp file says: The thread tried to read from or write to a virtual address for which it does not have the appropriate access 

We currently have a case with esri support and have been providing logs and any error messages so far.

If anyone has any ideas, please kindly share.

Thanks!

[JonEmch]

---

@BanchanaPandey wrote:

this machine has resources in terms of ram and cpu, page file space option is set to automatically grow.

---

Could you comment on whether or not RAM and CPU resources are meant to scale based on demand? Or are they static? If static, do they meet the minimum requirements as listed here: https://enterprise.arcgis.com/en/system-requirements/latest/windows/arcgis-server-system-requirement...

[BanchanaPandey]

Hi Jon,

Thanks for the information. Can you please tell me more on the scaling of RAM and CPU? 

I feel like they do. can i send the specs to you in private message?

Thanks!

[BanchanaPandey]

I am not finding an option to send it privately. I used to be able to do that earlier.

[BanchanaPandey]

@JonEmch 

The CPU and RAM are static.

Page file or virtual memory is dynamic as it is being set to Automatically manage paging file size for all drives.

Thanks!

[HenryLindemann]

Hi @BanchanaPandey,

I am going to take a stab at this and guess that you have more than 250 services on that system.

So if my guess is right there are a few possible solutions, I have seen the arcgisserver.exe crash when the server has a lot of services 500 and the server has low clock speed around 2.4 GHz, in this case the solutions was the reduce all the pooling minimum instances to 0 -You can up the pooling for critical services- and that stabilized the system.

The Second case that I had was a ArcGIS Server that had over 1000 services it generated the low memory error that you mentioned even though it had more than enough swap memory, the solution is shared instances , but this only works for services published out of pro so you will have to convert any ArcMap Services.

to implement case 2 

Go to ArcGIS Server Manager Site Settings and apply the settings as below 

Then go to all ArcGIS Pro Services and apply these settings 

Hope it helps 

Henry

[Todd_Metzler]

We've seen similar errors, although not exactly the same, as the number of services have grown in our environment [ArcGIS Enterprise 10.8.1 hosted on physical Windows server 2012R2, 2x server, 1x portal, 1x relational data store].  We implemented changes detailed in Esri Technical Article  and Microsoft Registry Heap Size  resulting in system stability improvement.

[ChristopherPawlyszyn]

If the Windows OS is reporting virtual memory exhaustion, that's usually a pretty reasonable diagnosis and is hopefully trivial to remedy. An important thing to keep in mind is that in the latest Windows Server operating systems (2016 and 2019), the page file growth is limited to the size of the volume it is on divided by eight. This means that for a 60GB C: volume, the default system-managed settings would restrict growth of the page file to 7.5GB.

With that in mind, you may want to work with your IT team to either define a static value for the page file (old-school recommendations range between 1.5-3x the amount of physical [or virtual] RAM on the machine), or move the page file to a volume with a larger size to allow for additional expansion of the virtual memory.

How to determine the appropriate page file size for 64-bit versions of Windows - Windows Client Management | Microsoft Docs
https://docs.microsoft.com/en-us/windows/client-management/determine-appropriate-page-file-size#syst...