ArcGIS Monitor not starting after upgrading to 2023.3.1

[TristanMcHardie]

After I upgraded from 2023.3 to 2023.3.1, ArcGIS Monitor wouldn't fully start. I could see processes running in Task Manager but I the UI wasn't responsive on ports 30443 or 30080 and nothing was bound to those ports when I checked using netstat.

This is how I fixed it, but I'm not completely confident this hasn't left things in an inconsistent state.

I uninstalled ArcGIS Monitor, backed up C:\Program Files\ArcGIS\Monitor and C:\Users\[serviceuser]\AppData\Local\ESRI\ArcGISMonitor\config-store-server, then reinstalled 2023.3.1. Things lit up after that, but of course I'd lost config-store-server which among other things includes user accounts, SSL certificates, etc.

I tried restoring config-store-server to the backed up version and it stopped working again. I took a copy of the fresh config-store-server, rolled back to a snapshot just prior to the uninstall, and replaced the original config-store-server with the fresh copy. This caused the original installation to start working, so the problem was narrowed down to config-store-server.

After some trial and error, I found that replacing .env and ssl\ssl.pfx with copies from the fresh install was enough to get the original installation working again. At that point I was prompted to go through /arcgis/monitor/setup to enter exiting credentials but everything was intact after that, though I will have to reconfigure the SSL certificate.

EDIT: I also found an .env.lock.STALE.STALE file in the original installation which didn't appear to have any impact on whether Monitor started or not.

So, the bug report in summary:

1. Something happened (or didn't happen) to config-store-server during the upgrade from 2023.3 to 2023.3.1.
2. Replacing .env and ssl\ssl.pfx with fresh copies is the minimum to get ArcGIS Monitor to start again.

And my questions:

Is this fix sufficient or will it have potentially made things inconsistent? Could it have been fixed more appropriately?

Thanks,
Tristan

[Tobias_Gerber]

We had a similar problem after upgrading from 2023.3. to 2023.3.1 and therefore opened a support case with Esri. The analysis showed that the .env file had completely incorrect content after the upgrade, so that the service could no longer start properly.

We took an old .env file from a backup (version 2023.3) and replaced the corrupt file with it. According to the Esri support, there is no schema change in the .env file between 2023.3 and 2023.3.1 so this should be - hopefully - a clean fix.

[EsriEvan]

Hi Tristan,

What you shared sounds like a scenario that affects some users upgrading to 2023.3.1, where, due to the size of the DB, upgrade's data model migrations take longer than expected, which leaves the server in a state where the user cannot connect until the migrations complete. Dev team is working on providing feedback in these cases so we can be more confident during upgrades. 

From your explanation, I think "toggling" the config-store is just disconnecting from the registered DB. The original config-store is probably fine, it's just triggering that data model upgrade and it's taking a while. If you contact support, they have some quick steps to validate this is the scenario and accelerate the upgrade.

Thanks,

Evan

[TristanMcHardie]

Thanks @EsriEvan and @Tobias_Gerber for your responses.

I'll reach out to our distributor to see if they can help.