Server service needs restarted every 7 days or so

[RandyBonds_Jr_]

About every seven days or so our services stop serving data and I have to restart the ARCGIS server service. 

this happened after the upgrade to 11.3. We have set the logs to debug and don’t see anything wrong.

just shooting the south there to see if anyone has ideas or suggestions. We’ve talked to support and they’ve been little to no help.

[berniejconnors]

We have regularly scheduled reboots. We have 3 ArcGIS Servers in a cluster and each night one Server is rebooted.  This helps prevent services from having your problem.  I have heard similar feedback from other ArcGIS Server admins.

[DavidPike]

What's the RAM usage like and number of services?  The ArcSOC processes associated to them can be quite hungry.

[RandyBonds_Jr_]

We’re at 30% or so. We checked that and CPU, they seem well. The CPU sits around 30-40%

[berniejconnors]

Schedule a reboot of the ArcGIS Server VMs once or twice a week and your problems with failing map services should be much less.  If not, you'll have to dig deeper.  You should record these events as they occur so you can measure any improvement you might see from the changes you make.

Bernie.

[AngusHooper1]

Where is the data located that the service is using with respect to the ArcGIS Server? It is common for a change to occur or other element (e.g. stale session) during the healthy uptime of the service. Once that service automatically recycles overnight it can then fall into a failed state as it cannot reach the data anymore.

If the issue is regular and consistent then that suggests some sort of automated operation occurring in the back end. For example, a database clean up that kills all sessions older than 3 days. If your services recycle every 7 days then this would become a problem.

[RandyBonds_Jr_]

The data lives on a high availability SQL Server, the data does not go offline. 

[RandyBonds_Jr_ · Accepted solution]

Wanted to provide an update to this ongoing issue. We MAY have solved it. 

Steps taken:

• Doubled the vCPUs used to 12
• Upped shared instance pool to match the change (ESRI recommends 50% of vCPU count)
• Removed some unused services
• Tuned other highly used services (mostly making them dedicated instead of shared)
• Found a backup job that was running weekly that may have caused slowness, but because of vCPU change, it's unclear if this is part of the issue.

Right now, I'm at 9 days uptime... here's hoping it's the end of this!

[berniejconnors]

@RandyBonds_Jr_ ,

All of those changes will definitely help.  Especially moving your highest traffic services to dedicated instances.  In my mind, that is the way "dedicated" and "shared" instances are intended to be used.

Bernie.

[RandyBonds_Jr_]

We had most of our services that way, a newer one that kind of took off was the real one that we made dedicated. 

Tuning is never ending, right?!