ArcGIS Monitor experiences recurring interruptions when collecting and reporting statistics from the ArcGIS Server cluster

WalterSimonazzi_VicPol

Hi team, @DerekLaw 

We have a 6‑machine ArcGIS Server cluster that is under heavy load, but it is also well resourced. We are not observing any CPU or memory pressure, RAM usage is consistently at or just below 80%, and CPU utilisation remains within acceptable limits.
One of our feature services, hosted in an Enterprise Geodatabase (SQL Server), is heavily used and is currently handling approximately 140 requests per second max. From an ArcGIS Server perspective, the platform is handling this load without issue.
However, ArcGIS Monitor is consistently failing for this particular ArcGIS Server cluster. We operate multiple ArcGIS Server clusters, and this issue appears isolated to the busiest one. When reviewing both Windows Event Logs and ArcGIS Server logs, we are not seeing any meaningful errors or indicators that explain the Monitor failures.
This raises the question:
Is ArcGIS Monitor known to struggle or fail when monitoring very busy ArcGIS Server clusters, even when system resources are not constrained? if so, is there a way to remediate this?

 

DerekLaw

Hi @WalterSimonazzi_VicPol,

Apologies for the late reply. I've been out of office on vacation for the last 2 weeks since the Esri Dev & Tech Summit ended.

> However, ArcGIS Monitor is consistently failing for this particular ArcGIS Server cluster.

It would be helpful if you could please elaborate on this statement. What do you mean by "failing"? 

Looking at the screenshots in your post, do you mean because the graphs have a status of "no data"? What happens when you run the inventory and metrics Observers manually for the ArcGIS Server component showing this behavior - does Monitor start to collect data for the ArcGIS Server component again?

There is some data collection, because the graphs do show some data, although for this one:

I agree and it does look like between 4 pm and 12 pm - Monitor stopped collecting data. Did something happen to the host during this time? Maybe patches were applied or perhaps a security scan was run that may have interfered with AM Agent collecting data and sending it back to AM Server? When you restart the AM Agent, does it start to collect data again?

> Is ArcGIS Monitor known to struggle or fail when monitoring very busy ArcGIS Server clusters, even when system resources are not constrained?

Nope, not to our knowledge. And this statement is pretty broad and suggests ArcGIS Monitor failure with no evidence, nor a valid reproducible case. Please contact Esri Tech Support so they could help investigate further. 

Hope this helps,