ArcGIS Server Log Error - Timeout: Pool Empty. Unable to Fetch a Connection

[RyanUthoff]

I am experiencing an error in the ArcGIS Server log files and I am wondering what it means and what is causing it. When I receive this error message, the feature services no longer work in the Portal web maps. Sometimes the error resolves itself in a few minutes, and sometimes I have to restart the ArcGIS Server service on the server.

It's happened a handful of times now and I would like to know what is causing this so I can resolve the issue. The error message from the log file is below. Anyone know what might be causing this?

[https-jsse-nio-6443-exec-223] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:100; busy:100; idle:0; lastwait:30000].

[GeoJosh]

Hi Ryan,

Run the following commands on the machine where Data Store is installed when the error message appears in the Server logs:

> cd C:\Program Files\ArcGIS\DataStore\tools\

> describedatastore.bat

What is returned for the number of connections to to Data store? If this number is greater than or approaching 150, you may want to consider increasing the maximum number of Data Store connections. You can do so using the following command:

> C:\Program Files\ArcGIS\DataStore\tools>changedbproperties.bat --max-connections 200 --store relational

-Josh

[RyanUthoff]

Thank you for your response. I already opened a support ticket with Esri regarding this issue and I was asked to do the same thing. It usually has about 15 connections to the Data Store, but quite awhile ago we changed the maximum number of connections from 150 to 300 since we do utilize the ArcGIS Data Store quite a bit (we have about 350 users). But I've never seen the number of connections anywhere near 150 or 300. Tech support and I are at the point where we need the error to happen again so we can see how many connections are being made to the database to confirm it isn't above the maximum allowed.

With that being said, I did some more investigating this morning and was able to find the previous time this error happened. Both times that this error happened, the CPU usage on the Data Store server (dedicated VM with only Data Store installed) spiked to 100%. It never spikes to 100% any other time, so it probably isn't a coincidence that we're receiving the error message the same time CPU is at 100%. Unfortunately, I can't see what caused the CPU to jump to 100%, but I will look the next time this happens.

[davehorsey1]

Hi @RyanUthoff. I appreciate this is asking you to cast your mind back quite a long time, but how were checking the number of connections to the datastore - was it using the Describedatastore tool? Thanks

[JervinAlbeso]

I experience the same issue on our Enterprise 10.8.1. Do we have any update on what might causing this?
We will try to run describedatastore once the issue occur again, however like what what Ryan said we rarely past 150 connections.

The datastore connections at my end has been adjusted to 200 connections.

Looking forward to some answers.