Monitored GeoEvent Servers disappearing regulary

[Tobias_Gerber]

We're monitoring three ArcGIS GeoEvent Servers with ArcGIS Monitor 2023.0 since several weeks and are experiencing the following strange behaviour: the registered component (Type "ArcGIS GeoEvent Server") is disappearing in ArcGIS Monitor regulary (although the GeoEvent Servers are still running indeed).

.

We noticed two different scenarios (the observers are set to 15 minutes for Discover and Metrics):

1. The GeoEvent Server component disappears and reappears 15 minutes later (occurs approx. 1 time a day)
2. The GeoEvent Server component disappears continuous. Only after a restart of the "ArcGIS Monitor Server"-Service it reappears again (occurs approx. 1 time a week)

It's difficult to recognize any pattern. Sometimes only 1 of our 3 GeoEvent Server is affected, sometimes all 3 at the same time. The point in time seems randomly distributed over the whole day. It seems that the monitoring process is somehow not "robust" enough for any potentially temporarily network issues/timeouts. We've monitored those GeoEvent Server already with the old ArcGIS Monitor 10.8 - not having any similar problems.

Logs entries for scenario 1:

At the time of disappearing I found those entries for the observer of the ArcGIS GeoEvent

• Warning with Code ETIMEDOUT and Message "Run failed."
• Severe with Code null and Message "["Timeout awaiting 'lookup' for 200ms"]"

Logs entries for scenario 2:

At the time of disappearing I found those entries for the observers of the ArcGIS GeoEvent

• Warning with Code null and Message "Run failed."
• Severe with Code null and Message " ["Response code 403 (Forbidden)"] "

After restarting the "ArcGIS Monitor Server"-Service - without having changed anything in the configuration! - the components are reappearing again.

Has anybody seen any similar behaviour? Is it possible to increase the waiting timeouts for the observers?

[DerekLaw]

Hi @Tobias_Gerber,

Can you please contact Esri Tech Support and open an issue so they can investigate this item further?

Thanks,

[Tobias_Gerber]

Hi @DerekLaw I'll open a support call.

[OscarFigueroaOD]

Hi Derek

Any luck getting this solved by help desk?  I am getting the same issue with the two geoevent observers.

I get the "Response code 403 (Forbidden)" error in the details.  Thanks

[Tobias_Gerber]

I have opened a support call and tried to isolate the problem together with Esri support. It is quite difficult because it is not easy to reproduce. With the adjustment of the time interval for the observers it has improved for a while - however, the problem was not solved in the end.
My next plan is to install an additional ArcGIS Monitor Agent locally on the GeoEvent Server - hoping that this will reduce connection timeouts and authentication problems.

[DerekLaw]

Thanks @Tobias_Gerber for sharing additional comments about your scenario.

@OscarFigueroaOD - Please contact Esri Tech Support and open an issue. Hopefully you can share more info with them so they can reproduce the problem. If we can reproduce the issue consistently, then we can look into a fix (if the issue is with ArcGIS Monitor).

Hope this helps,

[MatthewKelly_SP]

Hi - Im noticing the same behaviour. Was this ever resolved?

[Tobias_Gerber]

The problems described have disappeared in our environment since we installed additional ArcGIS Monitor Agents local on each GeoEvent Server.