What would cause all ArcGIS services to stop?

[ChrisSergent]

Upon restart of ArcGIS for Server all services went from starting to stopping. What might cause this?

[ChrisSergent]

Yes we manually restart ArcGIS for Server.

[PatrickJackson]

okay good to know Chris, lets see what the system resources look like when this happens again. Also check for me and see if the ArcGIS Server Service is set to "Automatic (delayed start)".

Thanks

[ChrisSergent]

It's not set to automatic or delayed start. We have to manually restart it. Our network staff has a security issue with setting that up, which is why we have to manually restart or start it.

[PatrickJackson]

Thanks for the updates Chris,

Just to clarify the behavior were seeing lets go through it step by step when the issue occurs.

At some point the machine has to be rebooted for whatever reason.

Then when it is up and running we Manually start the ArcGIS Server.

The published services begin to start up but then flip to stopping shortly after the ArcGIS Server starts up? 

The ArcGIS Server is then manually restarted and it starts up as normal.

Is this accurate to what we are seeing in the environment?

Please advise.

[ChrisSergent]

This appears to be accurate.

[PatrickJackson]

Thanks for confirming that Chris. We will need to see what the environment looks like when the machine has to be rebooted again. Before it gets rebooted can we set the logging to verbose on the ArcGIS Server?

You can do this by logging into the ArcGIS Server Manager > Click on Logs > change the "log filter" to verbose.

Hopefully we can get something useful when this occurs again in the logs in this way.

Post up the results when you can.

Thanks!

[ChrisSergent]

I was actually able to get the log file from last startup. Here's the pdf report: Dropbox - Error Report.pdf

[PatrickJackson]

I see several token errors and then several DBMS and SQL errors in those logs.

Is the database that is hosting content for these map services on this same machine?

Perhaps we need to look at how long it takes for the database to come up after a reboot. If the ArcGIS Server is coming up first and trying to pull data or check connections to the database before it is ready to go we might see errors like the ones in the logs.

This is where I would go next to troubleshoot.

[AlexPerez]

We were having the same problem.  Adding memory to the server solved the problem.  Our servers wouldn't give any sort of warning, they would just shut down when memory usage was above 95% and swap space got low.

[Kathleen_Crombez]

I know this is an older post, but we just ran into a similar issue where we could not get the services to start and they would be stuck in a "stopping" status. Most of the articles I read all point to low memory or network changes being the cause of this.

In our scenario we think it was related to new anti-virus software that was recently installed on the servers.

While researching this I ran across this article about excluding ArcGIS Enterprise directories from the virus scans.

FAQ: Which ArcGIS Enterprise directories should be excluded for security or antivirus software? 

Different antivirus and security software on the market and in computing environments sometimes detect ArcGIS Enterprise directories or components as a threat to the systems and networks with which they interact. When detected, the software may attempt to fix the ArcGIS Enterprise threat as it would any other, resulting in a partial or total loss of functionality for ArcGIS Enterprise.

To ensure the best functionality of ArcGIS Enterprise with the environment security, the following directories can be marked as excluded.

• Portal for ArcGIS
  • <installation drive>:\arcgisportal
  • <installation drive>:\Program Files\ArcGIS\Portal
• ArcGIS Server
  • <installation drive>:\arcgisserver
  • <installation drive>:\Program Files\ArcGIS\Server
  • <installation drive>:\Python27
• ArcGIS Data Store
  • <installation drive>:\arcgisdatastore
  • <installation drive>:\Program Files\ArcGIS\Datastore

This solution seems to have worked for us.

Hopefully this will help someone else in the future because we battled with this for over a month before we figured it out.