topic Re: Server service needs restarted every 7 days or so in ArcGIS Enterprise Questions

Server service needs restarted every 7 days or so

RandyBonds_Jr_ — Sat, 04 Jan 2025 19:42:18 GMT

About every seven days or so our services stop serving data and I have to restart the ARCGIS server service.

this happened after the upgrade to 11.3. We have set the logs to debug and don’t see anything wrong.

just shooting the south there to see if anyone has ideas or suggestions. We’ve talked to support and they’ve been little to no help.

Re: Server service needs restarted every 7 days or so

berniejconnors — Sun, 05 Jan 2025 14:01:48 GMT

We have regularly scheduled reboots. We have 3 ArcGIS Servers in a cluster and each night one Server is rebooted. This helps prevent services from having your problem. I have heard similar feedback from other ArcGIS Server admins.

Re: Server service needs restarted every 7 days or so

DavidPike — Sun, 05 Jan 2025 14:19:03 GMT

What's the RAM usage like and number of services? The ArcSOC processes associated to them can be quite hungry.

Re: Server service needs restarted every 7 days or so

RandyBonds_Jr_ — Sun, 05 Jan 2025 19:04:26 GMT

We’re at 30% or so. We checked that and CPU, they seem well. The CPU sits around 30-40%

Re: Server service needs restarted every 7 days or so

berniejconnors — Sun, 05 Jan 2025 22:53:03 GMT

Schedule a reboot of the ArcGIS Server VMs once or twice a week and your problems with failing map services should be much less. If not, you'll have to dig deeper. You should record these events as they occur so you can measure any improvement you might see from the changes you make.

Bernie.

Re: Server service needs restarted every 7 days or so

AngusHooper1 — Mon, 06 Jan 2025 02:28:03 GMT

Where is the data located that the service is using with respect to the ArcGIS Server? It is common for a change to occur or other element (e.g. stale session) during the healthy uptime of the service. Once that service automatically recycles overnight it can then fall into a failed state as it cannot reach the data anymore.

If the issue is regular and consistent then that suggests some sort of automated operation occurring in the back end. For example, a database clean up that kills all sessions older than 3 days. If your services recycle every 7 days then this would become a problem.

Re: Server service needs restarted every 7 days or so

RandyBonds_Jr_ — Mon, 06 Jan 2025 16:45:19 GMT

The data lives on a high availability SQL Server, the data does not go offline.

Re: Server service needs restarted every 7 days or so

RandyBonds_Jr_ — Mon, 13 Jan 2025 22:45:08 GMT

Wanted to provide an update to this ongoing issue. We MAY have solved it.

Steps taken:

Doubled the vCPUs used to 12
Upped shared instance pool to match the change (ESRI recommends 50% of vCPU count)
Removed some unused services
Tuned other highly used services (mostly making them dedicated instead of shared)
Found a backup job that was running weekly that may have caused slowness, but because of vCPU change, it's unclear if this is part of the issue.

Right now, I'm at 9 days uptime... here's hoping it's the end of this!

Re: Server service needs restarted every 7 days or so

berniejconnors — Tue, 21 Jan 2025 16:26:33 GMT

@RandyBonds_Jr_ ,

All of those changes will definitely help. Especially moving your highest traffic services to dedicated instances. In my mind, that is the way "dedicated" and "shared" instances are intended to be used.

Bernie.

Re: Server service needs restarted every 7 days or so

RandyBonds_Jr_ — Tue, 21 Jan 2025 16:44:19 GMT

We had most of our services that way, a newer one that kind of took off was the real one that we made dedicated.

Tuning is never ending, right?!

Re: Server service needs restarted every 7 days or so

berniejconnors — Tue, 21 Jan 2025 17:46:50 GMT

We just completed some special tuning for an event that we knew was going to greatly increase our traffic on one specific map service. We used some rules in Apache to identify incoming traffic for this map service and then split the load 50 / 50 across our 3 PROD servers and 3 TEST servers. Our TEST servers have the same specs as our PROD servers but TEST gets much less traffic. It worked perfectly and we experienced no problems with the additional traffic. We'll undo the special Apache config in a day or two.

Bernie.