Strange DataStore Connection Issue... then automatically fixes?

[JohnSteed1]

We were unable to see content in our WebApplications this morning.  Generic "The layer .... cannot be added to the map."

We looked at the logs and saw a ton of the following errors coming from both of our Servers (two in our deployment):

SEVERE Apr 13, 2021, 7:56:58 AM FATAL: remaining connection slots are reserved for non-replication superuser connections Rest

SEVERE Apr 13, 2021, 7:56:58 AM FATAL: sorry, too many clients already Rest

During this time the DataStore would not validate with the hosting Server.

And then it magically resolved (maps and WebApplications load content fine), but left us with constant logs from our hosting Server like this:

SEVERE

Apr 13, 2021, 8:14:22 AM

Update for the object was not attempted. Object may not exist.

Rest

Any ideas?

[berniejconnors]

This is the type of question I send directly to Esri's user support.  Your are paying for user support as part of your software licensing so you should use it.  I hope this community forum is not your primary source for support. 

It is helpful to the community to share information like this in the forum but to get the fastest solution to your problem you should contact Esri user support and then post your problem and the resolution from Esri here so we can all benefit.

Just my $0.02,

Bernie.

[JohnSteed1]

Hi Bernie-

I also sent this to Esri support, but sometimes I get better information here.

I'll make sure to provide feedback when this is resolved.

Thanks!

John

[berniejconnors]

Good to know John. I presumed you were also contacting Esri Support but I thought it was worth mentioning for the Community.
Bernie.

[AmberKelly]

Hi John,

Just wondering if you got any answers from Esri on this one?

Cheers

Amber

[JohnSteed1]

Hi @AmberKelly 
Thank you for your reply on this.
We did NOT get any answers on this.  It seemed to magically resolve.. which is unfortunate.

[AmberKelly]

Hi John,

Yes the "magic fixes" always leave me with uncertainty. Ill post here if I get any resolutions from Esri.

Thanks for replying!

Amber

[Vagish_Anand]

@JohnSteed1 @AmberKelly @berniejconnors 

ArcGIS Data Store uses postgresql as it's underlying database. The application(ArcGIS Data store) is designed in a way that it will allow max 150 connections to posgresql. You can check that by running describedatastore command line tool on data store machine. In the output you will see value for maxconnections as well as Number of connections which are active. Not all 150 connections will be allocated by ArcGIS Data Store to fetch data for ArcGIS Server. Some connection slots to postgresql are reserved for super users, in this case, ArcGIS Data Store applications. Hence the message,  SEVERE Apr 13, 2021, 7:56:58 AM FATAL: sorry, too many clients already.

If this is happening quite often then you can consider increasing the upper limit of Max Connections but remember that more connections will require a beefier machine in terms of memory and CPU processing power.