All Observers Stopped Reporting Metrics in Monitor 2024.1.1

[SusanHmel__DOIT_]

We noticed all of our alerts suddenly stopped over the holiday, and we're seeing that none of our observers are returning metrics. Agent status for all the hosts is ok. 

Getting an error in the ArcGIS Monitor logs for many of the observers for the agents (all of them, it seems - we have 38 hosts we monitor and I haven't checked all), the Observers for logs and inventory seem to be ok, but observers for metrics are not returning anything. 

The ArcGIS Monitor Log has this about every minute:

"Failed to process synthetic metrics results"

Plus a "Failed to process result for observer X" for each agent, 

I tried to restart the AM service on the server, and also restart a few different agent services on remote servers. The agents appeared offline then came back online so I don't think it's a connection issue, I tried unregistering one of our smaller sites and re-registering, same result, no metrics. 

We'll submit an Esri Tech Support request for this, but want to see if this issue has shown up anywhere else.

[SusanHmel__DOIT_ · Accepted solution]

Following up to my own post - we worked through and Esri Support request and were pointed to this bug:

https://support.esri.com/en-us/bug/unable-to-display-metrics-for-all-registered-components-bug-00017...

Basically our metrics count exceeded the possible values for the metrics sequence ID in the database. Changing the datatype to a BIGINT will give us a few more years of sequence values. I was able to follow the steps to make the PGSQL database updates and get the metrics going again.