ArcSOC.exe processes disappear: ArcGIS Enterprise Server 10.9.1

[AdamMiller4]

We recently started having issues with one of the servers in our HA 10.9.1 Enterprise server environment.  This server is paired with another as one site within our federated environment.  When the ArcGIS Server service is restarted the ArcGISServer.exe, javaw.exe, and multiple ArcSOC.exe services are started as normal.  However, after a couple minutes all of the ArcSOC processes disappear from the details pain within Windows Task Manager.  The ArcGISServer.exe and javaw.exe still seem to be running.  I have tried rebooting the server and stopping/restarting the ArcGIS Server Service and see the same results.   I am also finding repeated errors within the server logs similar to the following: <type="WARNING" code="7709" source="Server" process="4132" thread="1" methodName="" machine=* user="" elapsed="" requestID="">The Web Server was found to be stopped when it should have been started. Restarting it.</Msg>

Any ideas where to start first on troubleshooting?

Thanks!

[AngusHooper1]

This is a common symptom for a fundamental issue in your ArcGIS Server stack. If you have deployed to a windows environment, the next step would be to investigate the logs in Event Viewer for when you start/stop the service.

[AdamMiller4]

Angus I agree that is a good next step.  Here are a few of the events I see in the event log when I stop and start the ArcGIS Server Service. 

Error - Event ID 8198 - License Activation (slui.exe) failed with the following error code:

Information - Event ID 4, 5,6,7 (All with the following text)

The description for Event ID 6 from source ArcGIS Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

ArcGIS Server

The specified resource type cannot be found in the image file

I'm starting to decipher these different event ID's.

[Scott_Tansley]

I may be reading too much into this, but is one server working and one server not working?  Or are both servers affected in the same way?

[AdamMiller4]

You are reading into it correctly.  One of the servers in the site is working correct (although very strained) and the other is exhibiting this behavior.  

[JoshuaBixby · Accepted solution]

For certain ArcGIS Server issues, I have given up on trying to understand why and focus on what needs to be done to resolve the matter.  If the problem is recurring than knowing why is important, but I have found trying to understand why something broke is often chasing gremlins or ghosts, and Esri Support is often of little help.

Since you have a two machine site/cluster, I recommend unregistering the misbehaving ArcGIS Server, recycling the ArcGIS Server service on the misbehaving machine, and then re-registering it with the site.  I have found this often corrects random corruptions of an ArcGIS Server deployment on a machine.  As to why it happened, I stopped chasing that question a long time ago because I got more important tasks to focus my time on.

[AdamMiller4]

Unregistering, recycling, and joining back to the site seems to have worked!