We were unable to see content in our WebApplications this morning. Generic "The layer .... cannot be added to the map."
We looked at the logs and saw a ton of the following errors coming from both of our Servers (two in our deployment):
SEVERE Apr 13, 2021, 7:56:58 AM FATAL: remaining connection slots are reserved for non-replication superuser connections Rest
SEVERE Apr 13, 2021, 7:56:58 AM FATAL: sorry, too many clients already Rest
During this time the DataStore would not validate with the hosting Server.
And then it magically resolved (maps and WebApplications load content fine), but left us with constant logs from our hosting Server like this:
|SEVERE||Apr 13, 2021, 8:14:22 AM||Update for the object was not attempted. Object may not exist.||Rest|
This is the type of question I send directly to Esri's user support. Your are paying for user support as part of your software licensing so you should use it. I hope this community forum is not your primary source for support.
It is helpful to the community to share information like this in the forum but to get the fastest solution to your problem you should contact Esri user support and then post your problem and the resolution from Esri here so we can all benefit.
Just my $0.02,
ArcGIS Data Store uses postgresql as it's underlying database. The application(ArcGIS Data store) is designed in a way that it will allow max 150 connections to posgresql. You can check that by running describedatastore command line tool on data store machine. In the output you will see value for maxconnections as well as Number of connections which are active. Not all 150 connections will be allocated by ArcGIS Data Store to fetch data for ArcGIS Server. Some connection slots to postgresql are reserved for super users, in this case, ArcGIS Data Store applications. Hence the message, SEVERE Apr 13, 2021, 7:56:58 AM FATAL: sorry, too many clients already.
If this is happening quite often then you can consider increasing the upper limit of Max Connections but remember that more connections will require a beefier machine in terms of memory and CPU processing power.