nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Monitor 2024.0.1 - All monitored hosts are disconnected

EricJohnson-HDR

Monitor server is not connecting to any hosts. Under Admin\Logs there are recent entries with "Token is Expired", but not details as to which or what token is expired. All systems use same service account to register back to server. Another is "ArcGIS Monitor Agent is not ready"

Attempted to re-register with command line from host back to server, but stated validation exception. Assume due to there already being an existing machine with same name? If i disconnect the host, will i loose the historical data and start from ground zero?

I have restarted services, entire monitor server but still no luck. Restarted Postgres service and the existing DB seems to be good and healthy.

Out of options at this point. Let me know if you have some tricks.

Thanks

Encuentra más publicaciones etiquetadas con

Accepted answers

EsriEvan

Hi @SteveMcCarthy @EricJohnson-HDR, @GrahamWood-HWC

Thanks for sharing - I have been studying this and wanted to ask if you both wouldn't mind sharing a couple things:

1. Execute the following query in your Monitor Postgres and sharing the result. This will help me rule some things out.
SELECT
(SELECT last_value FROM metrics_id_seq) as metrics_id_seq,
(SELECT last_value FROM components_id_seq) as components_id_seq,
(SELECT last_value FROM observers_id_seq) as observers_id_seq;

2. Would you also confirm that the Agents in question show connected or disconnected? If they are disconnected, confirm we have attempted to re-register the Agent to Server using the --agent-id XYZ parameter?

Also wanted to share some thoughts related to other comments above:

- "Token expired logs seen" - these are probably just from the front end's natural token expiration / refresh and are beguine. Agent tokens do not expire.

- "Losing Components etc" - as long as the Agent is not "unregistered" the related Components and data would not be lost. If an Agent does somehow lost connection (as we are investigating here) one thing to try is go to the Agent and use the CLI "disconnect" and "register" using the --agent-id parameter to reregister the existing Agent and attempt to re-establish

All comments

DerekLaw

HI @EricJohnson-HDR,

> Monitor server is not connecting to any hosts. ... "ArcGIS Monitor Agent is not ready"

Did you also install AM Agent on the same machine as where you installed AM Server? I am wondering if you're upgrading from ArcGIS Monitor 2023 to 2024 (which has a different deployment architecture). Or was ArcGIS Monitor 2024 running normally, then this message appeared suddenly?

Side note: recommend you upgrade to the latest ArcGIS Monitor 2024.1.1 release - which includes many bug fixes over the 2024.0.1 release.

Hope this helps,

EricJohnson-HDR

Yes the Agent manager is on the same box as the server. It had been working fine, but seems the machine was patched and post reboot the AM service didn't start. So a week lapsed before i finally noticed it was down and then after the restart service, reboot of the server i was getting the no connectivity.

Seems all of the connections are still in place but for some reason not talking to the AM server.

Assume if I have to remove the current components and reconnect, I will loose all my Collections, Dashboards (analysis), tags etc?

Thanks

DerekLaw

Hi @EricJohnson-HDR,

Thanks for sharing the additional info. Just to be clear, you're running:

- AM Server 2024.0 and AM Agent 2024.0.1, right?

FYI in ArcGIS Monitor 2024.0.1 we fixed a bug related to AM Agents disconnecting from the AM Database when it becomes unavailable.

> Assume if I have to remove the current components and reconnect, I will loose all my Collections, Dashboards (analysis), tags etc?

If you unregister components in Monitor, your collections and analysis views will remain but will be empty. If you re-register them, then data collected previously will be unavailable.

Please open a case with Esri Tech Support so they can help investigate further.

EricJohnson-HDR

Both are 2024.0.1

Will pursue the ticket angle. Thanks

GrahamWood-HWC

Hi Eric

I've got a very similar issue that I've been working with tech support for a few months where i can only have metrics captured when I hit the run observer button in the Monitoring --> (arcgis server name)--> Settings-->Observers

I also have the token is expired message in logs. When I looked up the IP address it was my GIS workstations IP so I was assuming it was to do with the web interface.

Interestingly we're only now investigating the upgrade path after months of analysis of the issue.

SteveMcCarthy

I'm having a similar problem in AM 2024.1., All my agents stopped report to the AM server at the same time 4/5/2025 at 5:00 pm Central time. I have gone as far as to uninstall and reinstall my agents and they are still not communicating with server. I confirmed with my Security Team my ports are open, our Change Management says there were no changes that could affect AM.

Downtime.jpg

EsriEvan

Hi @SteveMcCarthy @EricJohnson-HDR, @GrahamWood-HWC

Thanks for sharing - I have been studying this and wanted to ask if you both wouldn't mind sharing a couple things:

EricJohnson-HDR

ok, i was able to reconnect using the command line with the --agent id parameter. So that is awesome. Looks like it reset my "names" i had established previously for my collections/analysis groups. Not sure what caused the agents to drop, but my money is on the recent Microsoft patches. As they have been the cause of a lot of drama in my environment.

Thanks for the reply and I am on my way back to normal.

EricJohnson-HDR

For others needing to get the Agent ID for your existing agents, you need to turn on that column in the Monitoring/Agent screen, as its off by default.

2025-04-10_11-05-45.PNG

EsriEvan

@EricJohnson-HDR - Thanks for sharing the latest. Acknowledging the OS updates, I am a bit troubled the Agents did not heal their own connection after some time 🤔. Will definitely flag this for future dev investigation.

SteveMcCarthy

@EsriEvan , @EricJohnson-HDR , @GrahamWood-HWC

My issue turn out to be caused by the Metric column getting to big for the Integer field and we need to convert it to BigInit. That resolved the issue right away. a big shout out to Premium Support, Mao did a great job on this.

TylerTheobald

Our 2024.1.1 AGM server was down for multiple weeks before we noticed. (I've since put a heartbeat monitoring in place")

Afterwards, all agents stayed disconnected until I had to register them all again with the Agent ID parameter. That was a bit of work for almost 100 agents.

Definitely something to get fixed in future versions.

Oiligriv

Hi, I'm in the same situation. I created this script in the first version, and I was able to re-register the VMs with little effort.

I'm posting it here, hoping it can help you in the future (it works on Windows).

# Script: Register-MonitorAgents.ps1

# Mappatura host -> agent-id
$agents = @{
    "prsitagsgp01.domain.it"  = 52
    "prsitagsgp02.domain.it"  = 54
    "prsitagsimg01.domain.it" = 53
    "prsitagsimg02.domain.it" = 55
    "prsitagsm01.domain.it"   = 56
    "prsitagsm02.domain.it"   = 57
    "prsitagspub01.domain.it" = 60
    "prsitagspub02.domain.it" = 61
    "prsitagsun01.domain.it"  = 42
    "prsitagsun02.domain.it"  = 43
    "prsitagsun03.domain.it"  = 44
    "prsitagsun04.domain.it"  = 45
    "prsitagsun05.domain.it"  = 46
    "prsitagsun06.domain.it"  = 47
    "prsitagsun07.domain.it"  = 49
    "prsitagsun08.domain.it"  = 48
    "prsitagsun09.domain.it"  = 50
    "prsitagsun10.domain.it"  = 51
    "prsitdata01.domain.it"   = 58
    "prsitdata02.domain.it"   = 59
    "prsithsd01.domain.it"    = 40
    "prsithsd02.domain.it"    = 41
    "prsitprt01.domain.it"    = 38
    "prsitprt02.domain.it"    = 39
    "prsitwebadp01.domain.it" = 36
    "prsitwebadp02.domain.it" = 37
}

# Parametri comuni
$serverUrl = "https://AGSPRMONITOR.domain.it:30443/arcgis"
$username  = "portaladmin"
$password  = "PortalPassword10@"
$exePath   = "C:\Program Files\ArcGIS\Monitor\Agent\bin\arcgis-monitor-agent.exe"

foreach ($vm in $agents.Keys) {
    $agentId = $agents[$vm]
    Write-Host ">>> Registrazione agente su $vm (Agent-ID: $agentId)" -ForegroundColor Cyan

    Invoke-Command -ComputerName $vm -ScriptBlock {
        param($exePath, $serverUrl, $username, $password, $agentId)

        & "$exePath" admin:system:register `
            --server-url $serverUrl `
            --username $username `
            --password $password `
            --agent-id $agentId
    } -ArgumentList $exePath, $serverUrl, $username, $password, $agentId
}

Virgilio

DerekLaw

Thanks for sharing your work @VirgilioPalmi !!

Únete a la Conversación

Los miembros registrados pueden publicar, seguir actualizaciones y más. ¿Nuevo aquí? Regístrate gratis.

Enlaces Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.