Service metrics disappearing

[Tobias_Gerber]

We have the following strange behaviour with our ArcGIS Monitor 2023.3.1

We monitor ArcGIS Server on 3 stages (Dev, Int, Prod). On all three sites a service A has been published and running for 50 days. When I look at the metrics for this service:

Available metrics for service A

• Dev Stage: 7 days
• Int Stage: 50 days
• Prod Stage: 2 days

I would expect the metrics for service A for 50 days for all three stages.

To make it even more confusing: While in one stage (Int) the metrics are constantly available over the entire period (50 days), in the other two stages the availability changes continuously, i.e. when I made a query 2 weeks ago for the metrics for service A (Dev stage), I got back the last 25 days, if I do the same today, it is only the last 7 days remaining.

This behaviour is similar for all services and not just one (service A is just an example)!

The metrics at the level of the entire site, however, are available for all 3 stages for the complete 180 days, which are set in the data retention policy settings (Administration=>General=>Database).

The Dev stage is monitored with a local ArcGIS Monitor agent on the ArcGIS Server machine. Int and Prod stage via the central agent on the ArcGIS Monitor Server itself.

Does anyone have a similar behaviour?

[GeoJosh]

Hi @Tobias_Gerber, sorry to hear you're experiencing this behavior. I think this is a known defect, which is addressed in the next release of Monitor: BUG-000165954 - The ArcGIS Monitor Services component IDs are recreated intermittently.

To verify that this is the defect responsible for behavior:

1. Take note of the component ID of one of the service components that you have noticed missing data for.

2. After the issue occurs again, check this ID to see if it has a different value.

Josh

[Tobias_Gerber]

Thanks @GeoJosh for the quick answer. I've noted the IDs of those services and will check the next days for any changes. 
I'll keep you updated.

[Tobias_Gerber]

Hello @GeoJosh 

Just 1 day later, the IDs for the services on one of our stage have already changed. Accordingly, I now only have the metrics for 1 day back for them.

I am glad to hear that there is already a known bug for this problem and that it should be fixed in the next release.

Is there any workaround in the meantime? Having only a 1 day history for the service metrics is quite a heavy functional limitation...

[GeoJosh]

Hi @Tobias_Gerber,

The underlying issue is that, for reasons unknown, the ArcGIS Server site sometimes has trouble cataloging its services which results in the site responding to requests from ArcGIS Monitor with unexpected payloads, which then results in the behavior you have noted. Because of this unpredictability on the ArcGIS Server side, there is no current workaround in Monitor. The fix in the next version of Monitor involves handling these unexpected responses more gracefully (a.k.a. not deleting service components every time an unexpected payload is processed), but it is still possible that ArcGIS Server responds in this unexpected way.

Josh

[Tobias_Gerber]

Thanks @GeoJosh for the honest answer. In that case we hope that the fix will be released soon and be robust enough to make this behaviour a very rare exception in the future.

[WhitneyLoy]

We are experiencing the same issue.