Mysterious failures of ArcGIS Server - Solved

[BurchettRobin]

Mysterious failures of ArcGIS Server.

1. Avista Utilities has begun building a set of Utility Network servers. In October 2020, ArcGIS Server on the Mapping servers failed intermittently, with no known cause. Some services would work and some would not, i.e. some layers would be missing and others would draw normally. We found that if we shut down one of the mapping servers, the other seemed to work normally, though users experienced it as slow because only one server was doing the work. We opened a case with ESRI, #02656978. After following the steps below, the problem was fixed.
2. Root cause:
1. Page File (aka virtual memory) size was too small, causing some ArcSOC.exe processes to shut down.
3. Solution:
1. Increase Page File size. We made this change to all boxes running ArcGIS Server, i.e. Mapping Server 1 & 2, Hosting Server 1 & 2, and Utility Network Server 1 & 2.
2. Steps are:
1. Go to Start Menu and click Settings.
2. Now type performance on search bar and select Adjust the appearance and performance of Windows.
3. A new window will open before you. Go to Advanced tab in it.
4. You’ll see a Virtual memory section in it. Click Change
5. Unselect the Automatically manage paging file size for all drives box on this window and click Custom size.
6. Enter 3967 the Initial Size box, and 10000 in the Maximum Size
7. Click Set
8. Do the same for the D drive.
9. Click Set
10. Click OK to save the changes.
3. We monitored the ArcSOC.exe processes on the mapping servers. They were almost all below 200,000k.
4. Additional details:
1. The techs (especially Aldo) looked at a lot of our log files. Aldo wrote:

I wanted to provide a quick recap of our call and troubleshooting from earlier today. We ended up finding several logs in both the NW0478 and NW0479 machines indicating low Virtual Memory:

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: javaw.exe (4888) consumed 968736768 bytes, ArcSOC.exe (29376) consumed 513667072 bytes, and ArcSOC.exe (24104) consumed 501309440 bytes.

At the moment things have been stable with just 479 on. There are no arcsocs taking alot of memory. The max memory the biggest arcsoc takes up is around 130 mb vs the almost 500 mb shown in the event viewer error above.

Now the low virtual memory could be the cause of the intermittent issues you guys experienced earlier when both machines were up. To alleviate the low virtual memory issue we ran through the steps in this article to change the paging file size of windows: https://support.esri.com/en/technical-article/000002245

We Verified on both mapping servers that the paging file settings were set to automatic with a limit of 3987 mb.

Changed the paging file setting for both machines and on both their C and D drives to have a minimum of 3987 mb and a maximum of 10000 mb.

Turned on the second server and verified things were running smoothly with no spike in performance nor did any arcsocs spring up in memory usage.