Mysterious failures of ArcGIS Server.
I wanted to provide a quick recap of our call and troubleshooting from earlier today. We ended up finding several logs in both the NW0478 and NW0479 machines indicating low Virtual Memory:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: javaw.exe (4888) consumed 968736768 bytes, ArcSOC.exe (29376) consumed 513667072 bytes, and ArcSOC.exe (24104) consumed 501309440 bytes.
At the moment things have been stable with just 479 on. There are no arcsocs taking alot of memory. The max memory the biggest arcsoc takes up is around 130 mb vs the almost 500 mb shown in the event viewer error above.
Now the low virtual memory could be the cause of the intermittent issues you guys experienced earlier when both machines were up. To alleviate the low virtual memory issue we ran through the steps in this article to change the paging file size of windows: https://support.esri.com/en/technical-article/000002245
We Verified on both mapping servers that the paging file settings were set to automatic with a limit of 3987 mb.
Changed the paging file setting for both machines and on both their C and D drives to have a minimum of 3987 mb and a maximum of 10000 mb.
Turned on the second server and verified things were running smoothly with no spike in performance nor did any arcsocs spring up in memory usage.