Monitor 2024.0.1 - All monitored hosts are disconnected

[EricJohnson-HDR]

Monitor server is not connecting to any hosts.  Under Admin\Logs there are recent entries with "Token is Expired", but not details as to which or what token is expired. All systems use same service account to register back to server.  Another is "ArcGIS Monitor Agent is not ready"

Attempted to re-register with command line from host back to server, but stated validation exception.  Assume due to there already being an existing machine with same name?  If i disconnect the host, will i loose the historical data and start from ground zero?

I have restarted services, entire monitor server but still no luck.  Restarted Postgres service and the existing DB seems to be good and healthy.

Out of options at this point.  Let me know if you have some tricks.

Thanks

[DerekLaw]

HI @EricJohnson-HDR,

> Monitor server is not connecting to any hosts. ... "ArcGIS Monitor Agent is not ready"

Did you also install AM Agent on the same machine as where you installed AM Server? I am wondering if you're upgrading from ArcGIS Monitor 2023 to 2024 (which has a different deployment architecture). Or was ArcGIS Monitor 2024 running normally, then this message appeared suddenly? 

Side note: recommend you upgrade to the latest ArcGIS Monitor 2024.1.1 release - which includes many bug fixes over the 2024.0.1 release.

Hope this helps,

[EricJohnson-HDR]

Yes the Agent manager is on the same box as the server.  It had been working fine, but seems the machine was patched and post reboot the AM service didn't start.  So a week lapsed before i finally noticed it was down and then after the restart service, reboot of the server i was getting the no connectivity.

Seems all of the connections are still in place but for some reason not talking to the AM server. 

Assume if I have to remove the current components and reconnect, I will loose all my Collections, Dashboards (analysis), tags etc?

Thanks

[DerekLaw]

Hi @EricJohnson-HDR,

Thanks for sharing the additional info. Just to be clear, you're running:

- AM Server 2024.0 and AM Agent 2024.0.1, right?

FYI in ArcGIS Monitor 2024.0.1 we fixed a bug related to AM Agents disconnecting from the AM Database when it becomes unavailable. 

> Assume if I have to remove the current components and reconnect, I will loose all my Collections, Dashboards (analysis), tags etc?

If you unregister components in Monitor, your collections and analysis views will remain but will be empty. If you re-register them, then data collected previously will be unavailable. 

Please open a case with Esri Tech Support so they can help investigate further.

[EricJohnson-HDR]

Both are 2024.0.1

Will pursue the ticket angle.  Thanks

[GrahamWood-HWC]

Hi Eric

I've got a very similar issue that I've been working with tech support for a few months where i can only have metrics captured when I hit the run observer button in the Monitoring --> (arcgis server name)--> Settings-->Observers 

I also have the token is expired message in logs. When I looked up the IP address it was my GIS workstations IP so I was assuming it was to do with the web interface.

Interestingly we're only now investigating the upgrade path after months of analysis of the issue.

[SteveMcCarthy]

I'm having a similar problem in AM 2024.1., All my agents stopped report to the AM server at the same time 4/5/2025 at 5:00 pm Central time.  I have gone as far as to uninstall and reinstall my agents and they are still not communicating with server.  I confirmed with my Security Team my ports are open, our Change Management says there were no changes that could affect AM.

[EsriEvan · Accepted solution]

Hi @SteveMcCarthy @EricJohnson-HDR, @GrahamWood-HWC 

Thanks for sharing - I have been studying this and wanted to ask if you both wouldn't mind sharing a couple things:

1. Execute the following query in your Monitor Postgres and sharing the result. This will help me rule some things out.
SELECT
(SELECT last_value FROM metrics_id_seq) as metrics_id_seq,
(SELECT last_value FROM components_id_seq) as components_id_seq,
(SELECT last_value FROM observers_id_seq) as observers_id_seq;

2. Would you also confirm that the Agents in question show connected or disconnected? If they are disconnected, confirm we have attempted to re-register the Agent to Server using the --agent-id XYZ parameter?

Also wanted to share some thoughts related to other comments above:

- "Token expired logs seen" - these are probably just from the front end's natural token expiration / refresh and are beguine. Agent tokens do not expire.

- "Losing Components etc" - as long as the Agent is not "unregistered" the related Components and data would not be lost. If an Agent does somehow lost connection (as we are investigating here) one thing to try is go to the Agent and use the CLI "disconnect" and "register" using the --agent-id parameter to reregister the existing Agent and attempt to re-establish

[EricJohnson-HDR]

ok, i was able to reconnect using the command line with the --agent id parameter. So that is awesome.  Looks like it reset my "names" i had established previously for my collections/analysis groups.  Not sure what caused the agents to drop, but my money is on the recent Microsoft patches.  As they have been the cause of a lot of drama in my environment.  

Thanks for the reply and I am on my way back to normal.

[EricJohnson-HDR]

For others needing to get the Agent ID for your existing agents, you need to turn on that column in the Monitoring/Agent screen, as its off by default.