Error connecting to monitor database after 2023.3 (2023.3.1) upgrade

[FraserHand]

Hi there,

We just tried to upgrade to 2023.3 and the database connection dropped. In ArcGIS Monitor db admin we had this:

clicking on the configure just returned a 500 error (db connection error in the json), so I had to clear the db settings in the .env file to get the software to allow a db connection to be registered.

On trying to reregister the db we had the same error described here:

https://community.esri.com/t5/arcgis-monitor-questions/error-connecting-to-monitor-database/td-p/125...

with the Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? error.

This looks to be something in node.js so I tried updating to 2023.3.1 to see if that cleared it. I now get a timeout trying to reregister the db

E_SERVER_TIMEOUT: Request timed out after 60000 ms

and then this if I reattempt

RuntimeException: Migrations are locked. Make sure you are not multiple migration scripts or delete `agm_schema_lock` table manually

Dropping the table and rerunning the db register just returns to

E_SERVER_TIMEOUT: Request timed out after 60000 ms

The connection to the db is fine as I've tested via pgadminb / heidi etc.

Restarting services does not help.

Is there a way to clear this and reregister the db?

Many thanks

Fraser

[FraserHand]

EDIT: As a test I created a new db from scratch in the same postgresql instance and attempted to register that - no issues connecting creating the schema and registering with ArcGIS Monitor - so it looks to be something when trying to migrate the schema perhaps?

[FraserHand]

EDIT EDIT: I was wondering if it was the HTTP_REQUEST_TIMEOUT=60000 setting in the env file - so I reconnected the original db to see if I could get the error back and then tweak the setting. This time the original db reconnected. Would be good know what happened and why switching dbs reset things.

Thanks!

[DerekLaw]

Hi @FraserHand,

I'm sorry to hear that you've encountered an issue with the ArcGIS Monitor 2023.3.1 patch. Can you please contact Esri Tech Support so they can collect more information and help investigate the issue?

[TimLehn]

I am having the same issue and would rather not create a whole new database.  Was wondering if there is a way to break the connections to allow for the connection to be reestablished?  When you click on the Configure database it goes nowhere.

[AustinMcCleary]

I also had this problem after running the upgrade. 

I ultimately went into it's .env file (C:\users\serviceaccountmonitorrunsunder\appdata\local\esri\arcgismonitor) and changed the time out to an astronomical number. 12000000

I saw in the schema tables there was a locked table - when trying to connect originally it was timing out. Like the error states in the OP.

I think in the upgrade process something happened and it didn't get to complete. So when it restarted it was half done. Changing the timeout seemed to have allowed it to finish what hadn't been completed is my guess.

After monitor finally connected I was able to reset the timeout back to default and the database and configurations were all good.

[TonyAmar1]

What we had to do to resolve the connection issue was to remove the encrypted password from the .env file for Monitor C:\Users\...\AppData\Local\ESRI\ArcGISMonitor\config-store-server (or wherever you have installed it). Restarting Monitor then allows the connection to return and asks you to enter a password for the database. Dropping the `agm_schema_lock` table then allowed the connection to work as normal. This is what we found that seemed to resolve the connection issue.

[mgress12]

This process worked for us upgrading from 2023.2 to 2023.3.1 and experiencing the database connection issue.