Enterprise database connections dropping

[Ona_Tim]

I'm not sure if this is the right forum but, I am having an issue with enterprise Database connections dropping to my computer. If I work with ArcPro on the server the connections do not drop. We also made a connection directly from my computer using pgAdmin and will not lose connection even when I loose it in Pro.  

My system is as follows:

1. ArcGIS Enterprise 11.2 Single machine deployment. Server has 32GB of memory
2. ArcgisPro 3.3.0
3. Postgres 15.4 Downloaded from ESRI and on a separate server from enterprise. Server has 12 cores and 32GB of memory
4. Both servers are running Windows Server 2022 datacenter Version 21H2.

When the connection drops I get the following message:

Underlying DBMS error [no connection to the server::SQLSTATE=å] [**unspecified**.**unspecified**.GDB_Items][STATE_ID = 0]

X's replace actual names.

Working with ESRI we found the following information in the logs :

"An existing connection was forcibly closed by the remote host" where host is the PostgreSQL Server and "FATAL: terminating connection due to administrator command". The following PostgreSQL documentation provides more information about the same (https://www.postgresql.org/message-id/5665.1202148155%40sss.pgh.pa.us [postgresql.org]).

At this time IT says it is not a fire wall issue as there are no rules setup for dropping connections. We are at a loss as to what the culprit is.

I will also be posting this in PostgreSQL community to get their input.

Thank you for your help with this.

[George_Thompson]

How are you accessing the Postgres EGDB, via VPN or directly in your office?

Does the timeout happen after a specific amount of time, like 15 or 30 minutes?

Does it drop the connection when in the middle of interacting with data or after a period of "inactivity"?

What are the resources on the Postgres VM reporting when the issue occurs? RAM / CPU utilization at that time?

[Ona_Tim]

George,

In our office through standard geodatabase connection in pro.

If Idle it seems like 20 minutes tops.

I have had it drop while working in the database too. I'm not sure how long I was working in it when it happened.

I have the RAM/CPU Utilization up right now and will take a screen shot when disconnects.  I have attached is screen shot when I opened the project.

[Ona_Tim]

Here is the screen shot after loosing connection.

[George_Thompson]

Ok, thanks for those items it does not appear to be resource related.

I am wondering if the PostgreSQL DB is running out of available connections?

I cannot find what I am looking for in order to check the # connections allowed.

[Ona_Tim]

Here are the logs from ESRI highlighting, in yellow, the issues. It seems the authentication for the password for the postgres user is failing. 

Edit:  I forgot to mention that we are not using the postgres user to connect to the GDB.

[ISM]

Hi!

I have the same issue and was wondering if you were able to resolve it.
Any help to resolve the problem would be greatly appreciated.

Thanks!

[Ona_Tim]

ISM, we have not yet found a solution to this.

[BenjiMillard]

I am having the same problems and I believe they are leading to ArcGIS Server crashing. I'm currently looking at the TCP Keepalive settings but haven't moved forward with any experimentation. 

https://www.postgresql.org/docs/current/runtime-config-connection.html

[BenjiMillard · Accepted solution]

My suspicions were confirmed about the Keep Alive settings of PostgreSQL. After PostgreSQL 14 there were changes made to the way that PostgreSQL connections were made and kept open that close connections in unexpected ways if you're used to the older versions.

Some settings that should change in postgres.conf are:

tcp_keepalives_idle=120

tcp_keepalives_interval=20

tcp_keepalives_count=6

# 5 second interval in milliseconds

client_connection_check_interval=5000

And my ArcGIS Server connection is no longer crashing regularly.