ArcGIS Enterprise 11.5 – Post Upgrade Intermittent Server Site Instability (Multi-Machine Site)

Question

Hi all,

I wanted to share some observations and see if others have experienced similar behaviour after upgrading to ArcGIS Enterprise 11.5, particularly in a multi-machine ArcGIS Server site. It has been a challenge understanding why we are now seeing stability issues, especially since the environment was stable for a long time on 11.1 without any notable problems.

This is a single logical ArcGIS Server site backed by two machines that are configured similarly, with the config store hosted on a Windows network share using a UNC path in Azure. Since the upgrade, we have been experiencing intermittent stability issues where one server machine appears to operate normally while the other experiences repeated CPU spikes, with java.exe frequently reaching between 70 and 100 percent utilization.

This behaviour is more noticeable during business hours and seems to stabilize later in the day. During these periods, the affected machine appears to drop out of the site, and ArcSOC behaviour becomes inconsistent, where processes ramp up during startup and then unexpectedly drop to zero. When this happens, services may still show as started but are not actually functioning correctly.

We have attempted to restart the ArcGIS Server service, but this does not consistently resolve the issue, and in some cases, ArcSOC processes fail to come back up at all. In most situations, a full VM reboot helps. I have also seen ArcSOC processes recover on their own, but this is inconsistent and not something we can rely on.

We have ruled out memory pressure, as utilization is generally between 40 and 60 percent. Disk space and service account permissions have also been validated and do not appear to be contributing factors. What is particularly concerning is that although this is a multi-machine site, the behaviour is not balanced. One machine consistently shows high CPU usage and instability while the other remains stable, which was not the case before the 11.5 upgrade.

Given this, I wanted to ask if anyone has experienced similar CPU instability on ArcGIS Server 11.5 in a multi-machine site, or has seen ArcSOC processes that do not align with what ArcGIS Server Manager is showing. For example, minimum instances may show as 0 or 1, but more processes are running on the machine for that service. I am also wondering if there are any known issues related to high java.exe CPU usage or machines intermittently dropping from a site, and whether there are any considerations with hosting the config store on a UNC path in Azure, specifically for 11.5.

Thank you in advance for any insights.

JCGuarneri · Answer

I hope you're not still having issues, but if you're using web adaptor, there's a stability patch you need to install. Things get pretty unreliable without it.

https://support.esri.com/en-us/patches-updates/2025/arcgis-web-adaptor-iis-stability-patch

JoshuaBixby · Answer

Is it always the same machine becoming unstable?  Since this is a multi-machine site, I would put one of the machines into maintenance mode for a day or two or three and see if the site remains stable with just one machine.  If the same machine becomes unstable now, drop that one into maintenance mode first.  Then swap which one is in maintenance mode.  If the site is stable when only running on one machine but highly unstable when only running on the other, you know that something likely went wrong with the upgrade process on that one machine.  At that point, chasing gremlins is seldom worth it, just drop the one machine from the site, uninstalled and reinstall software, and then re-join it back to site.

SH013 · Answer

Yes, I have been studying the logs closely since then. There are log entries that seem to line up with the behaviour we are seeing. What stands out most are intermittent SEVERE 7570 messages such as “Failed to construct instance of service" or "ArcSOC not bounded to RMI Registry,” which appear to point to a service instance failing during initialization rather than a normal request failure. In our case, these tend to show up around restart or recycle periods, often overnight, and the service may appear fine later, which makes me think the initial startup attempt failed but a later retry succeeded once conditions stabilized. I also suspect that during startup, which I have seen when I manually restart the services, the server seems to struggle, and CPU usage tends to spike up and down in unusual ways.

I am currently trying to offset instance recycling so that services are not all reinitializing at the same time. My thinking is that reducing that startup wave may help, especially for services with higher minimum instances. Even so, after an ArcGIS Server service restart or machine reboot, the site still seems to struggle to bring services back online cleanly, with some failing again during or shortly after initialization.

What I am trying to understand is why this has become an issue now. We did not see this kind of stability problem in 11.1, and the environment was stable for a long time. System resource consumption also appears to be fairly consistent overall. Since upgrading to 11.5, however, we have been dealing with intermittent stability issues over the last couple of weeks, including servers dropping from the site and services crashing. So I am trying to understand whether I am onto something with the initialization timing and instance startup angle, or whether 11.5 is simply behaving differently than 11.1 in a way that others have also observed.