topic Re: What would cause all ArcGIS services to stop? in ArcGIS Enterprise Questions

What would cause all ArcGIS services to stop?

ChrisSergent — Tue, 14 Apr 2015 14:10:29 GMT

Upon restart of ArcGIS for Server all services went from starting to stopping. What might cause this?

Re: What would cause all ArcGIS services to stop?

AaronKreag — Tue, 14 Apr 2015 17:00:02 GMT

We have 10.2.2.....running 60 map services...min 2 and max 25. For whatever reason, almost every time we reboot the application box and it will start the services and then shut them all down and then start again...all over like a 15 minute period. We see other odd behavior with this 10.2.2 install. Our version 10sp5 seemed more stable.

Re: What would cause all ArcGIS services to stop?

PatrickJackson — Tue, 14 Apr 2015 19:09:38 GMT

Greetings,

Can you please advise the following:

What version of ArcGIS for Server we are looking at for this?

What is the OS that is being used?

How many published services are running on this machine?

What do the System Resources look like as you watch this occur? (CPU, Memory, etc.)

One thing that may help with better startup after a reboot of the Machine is to set the ArcGIS Server Service in Windows Services to Automatic (Delayed Start). This will wait for other services to start up and then start the ArcGIS Server up.

Thanks!

Re: What would cause all ArcGIS services to stop?

AaronKreag — Tue, 14 Apr 2015 19:18:49 GMT

Patrick-

What version of ArcGIS for Server we are looking at for this? – 10.2.2

What is the OS that is being used? - Windows 2012

How many published services are running on this machine? - 52

What do the System Resources look like as you watch this occur? (CPU, Memory, etc.) - 16% CPU and 42% RAM

Aaron

Re: What would cause all ArcGIS services to stop?

DennisSmith — Wed, 15 Apr 2015 15:24:37 GMT

May I ask why you are rebooting the ArcGIS Server? Typically this behavior happens when you are set to Automatic (Delayed Start) and all of the services are trying to spin up to run. Also how many of your services are set to minimum instance of 1 (or More)? These are the services that are going to start-up with the server.

Re: What would cause all ArcGIS services to stop?

AaronKreag — Wed, 15 Apr 2015 15:48:53 GMT

Does ESRI recommend not rebooting servers?

Rebooting of an ESRI server can happen for a list of reasons....but maintenance or updates are the norm, or as a last resort when the application stops responding and ESRI support asks if we can reboot the server in the middle of the day, etc etc etc.

I would suggest that if rebooting is not recommended...1. it be documented and 2. the software be more stable. For the record, every ESRI course I have ever taken and every ESRI tech person I have talked to in the last 6 years says "rebooting AGS weekly ensures the best health and performance"...clearing hung processes, clearing temp directories, freeing up RAM, etc.

I think all of our services are set to 2 min. We can change to 1 min but 0 min causes "service not available" errors in the web map on our lessor used services until it gets woke up and the SOC process starts, this causes end users to call and report issues and errors that go away with a refresh after a few minutes but clog the phone lines and email. So setting to 0 is not an option.

Not sure about the original poster.....

Aaron

Re: What would cause all ArcGIS services to stop?

ChrisSergent — Wed, 15 Apr 2015 15:54:07 GMT

For us, people were not able to connect to our data as web services were not displaying the data.

Re: What would cause all ArcGIS services to stop?

DennisSmith — Wed, 15 Apr 2015 16:03:19 GMT

Chris,

Can you please advise the following:

What version of ArcGIS for Server we are looking at for this?

What is the OS that is being used?

How many published services are running on this machine?

What do the System Resources look like as you watch this occur? (CPU, Memory, etc.)

Re: What would cause all ArcGIS services to stop?

ChrisSergent — Wed, 15 Apr 2015 16:21:37 GMT

ArcGIS for Server version: 10.2

OS: Windows Server 2008 R2 Standard

Published Services: There are 65 published services; this includes utilities and System

System resources: Not sure on that one, but if and when we have to do it again, we will make a note of it, but we need to keep our services running

Re: What would cause all ArcGIS services to stop?

PatrickJackson — Wed, 15 Apr 2015 16:35:36 GMT

Thanks for those details Chris,

How is the ArcGIS Server set to start up when the Server is coming up from the reboot? This would be configured in Windows Services. Is the ArcGIS Server set to "Automatic" or "Automatic (delayed start)"?

Based on your original post it sounds like the services stop and then never start up until the ArcGIS Server is manually restarted. Is this correct?

Re: What would cause all ArcGIS services to stop?

ChrisSergent — Wed, 15 Apr 2015 16:43:37 GMT

Yes we manually restart ArcGIS for Server.

Re: What would cause all ArcGIS services to stop?

PatrickJackson — Wed, 15 Apr 2015 17:02:19 GMT

okay good to know Chris, lets see what the system resources look like when this happens again. Also check for me and see if the ArcGIS Server Service is set to "Automatic (delayed start)".

Thanks

Re: What would cause all ArcGIS services to stop?

ChrisSergent — Wed, 15 Apr 2015 18:34:03 GMT

It's not set to automatic or delayed start. We have to manually restart it. Our network staff has a security issue with setting that up, which is why we have to manually restart or start it.

Re: What would cause all ArcGIS services to stop?

PatrickJackson — Wed, 15 Apr 2015 18:49:08 GMT

Thanks for the updates Chris,

Just to clarify the behavior were seeing lets go through it step by step when the issue occurs.

At some point the machine has to be rebooted for whatever reason.

Then when it is up and running we Manually start the ArcGIS Server.

The published services begin to start up but then flip to stopping shortly after the ArcGIS Server starts up?

The ArcGIS Server is then manually restarted and it starts up as normal.

Is this accurate to what we are seeing in the environment?

Please advise.

Re: What would cause all ArcGIS services to stop?

ChrisSergent — Thu, 16 Apr 2015 15:29:20 GMT

This appears to be accurate.

Re: What would cause all ArcGIS services to stop?

PatrickJackson — Thu, 16 Apr 2015 15:37:26 GMT

Thanks for confirming that Chris. We will need to see what the environment looks like when the machine has to be rebooted again. Before it gets rebooted can we set the logging to verbose on the ArcGIS Server?

You can do this by logging into the ArcGIS Server Manager > Click on Logs > change the "log filter" to verbose.

Hopefully we can get something useful when this occurs again in the logs in this way.

Post up the results when you can.

Thanks!

Re: What would cause all ArcGIS services to stop?

ChrisSergent — Thu, 16 Apr 2015 16:05:33 GMT

I was actually able to get the log file from last startup. Here's the pdf report: Dropbox - Error Report.pdf

Re: What would cause all ArcGIS services to stop?

PatrickJackson — Thu, 16 Apr 2015 16:14:38 GMT

I see several token errors and then several DBMS and SQL errors in those logs.

Is the database that is hosting content for these map services on this same machine?

Perhaps we need to look at how long it takes for the database to come up after a reboot. If the ArcGIS Server is coming up first and trying to pull data or check connections to the database before it is ready to go we might see errors like the ones in the logs.

This is where I would go next to troubleshoot.

Re: What would cause all ArcGIS services to stop?

AlexPerez — Thu, 16 Apr 2015 18:10:18 GMT

We were having the same problem. Adding memory to the server solved the problem. Our servers wouldn't give any sort of warning, they would just shut down when memory usage was above 95% and swap space got low.

Re: What would cause all ArcGIS services to stop?

Kathleen_Crombez — Wed, 02 Jan 2019 16:44:01 GMT

I know this is an older post, but we just ran into a similar issue where we could not get the services to start and they would be stuck in a "stopping" status. Most of the articles I read all point to low memory or network changes being the cause of this.

In our scenario we think it was related to new anti-virus software that was recently installed on the servers.

While researching this I ran across this article about excluding ArcGIS Enterprise directories from the virus scans.

FAQ: Which ArcGIS Enterprise directories should be excluded for security or antivirus software?

Different antivirus and security software on the market and in computing environments sometimes detect ArcGIS Enterprise directories or components as a threat to the systems and networks with which they interact. When detected, the software may attempt to fix the ArcGIS Enterprise threat as it would any other, resulting in a partial or total loss of functionality for ArcGIS Enterprise.

To ensure the best functionality of ArcGIS Enterprise with the environment security, the following directories can be marked as excluded.

Portal for ArcGIS
- <installation drive>:\arcgisportal
- <installation drive>:\Program Files\ArcGIS\Portal
ArcGIS Server
- <installation drive>:\arcgisserver
- <installation drive>:\Program Files\ArcGIS\Server
- <installation drive>:\Python27
ArcGIS Data Store
- <installation drive>:\arcgisdatastore
- <installation drive>:\Program Files\ArcGIS\Datastore

This solution seems to have worked for us.

Hopefully this will help someone else in the future because we battled with this for over a month before we figured it out.