Server 10.3.1 error "An existing connection was forcibly closed by the remote host"

[BugPie]

I'm seeing an error in our Server logs since we have upgraded to v 10.3.1 about 2 months ago. I have not been able to locate a specific time, layer, or service for this error. It seems to happen at random.The error reads as follows, the bold being our enterprise specific names that I substituted.

"There is an error during the draw Layername (1.) Failure to access the DBMS server [08S01:[Microsoft][SQL Server Native Client 11,(2.) 0]TCP Provider: An existing connection was forcibly closed by the remote host,(3.) ] [dbname,(4.) DBO,(5.) FeatureName]."

We have our enterprise gdb in a SQL environment version 2008r2

the server OS for that server is Windows Server 2008r2 SP1

Our GIS server is on a machine running Windows Server 2008r2 SP1

GIS Server Version 10.3.1

Does this error mean the DBMS and or server has closed the connection with the GIS server? I have found that the GIS server has both NativeClient 10 and 11 installed for what it's worth. 

Should I have our network guy help me with this, or is it a SQL or ESRI specific problem I am seeing.  Hopefully just a simply connection timing setting I can adjust.

whatcha thinking out there?!

[DEWright_CA]

I have found this could be tied to another issue; one that results in a message like this:

Underlying DBMS error [[Microsoft][SQL Server Native Client 11.0][SQL Server]The app domain with specified version id (99) was unloaded due to memory pressure and could not be found.] [<featurelayer name>]

This is caused by a memory contention issue; where the SQL DB hosting machine does not have enough RAM between the running OS and the instance of SQLServer running; so when there is the contention; Microsoft fires the .Net GarbageCollection flushing the SQL Spatial Types libraries from memory and there by recycling the connections accessing them.

Try looking deeper at your SQL Server logs for the time frame where you had that error logged to you and see what you see in the server side of the conneciton.

[BugPie]

Hi David, thanks for your insight on this one. I took your advice and dug into the SQL logs a bit more and did not see anything explicitly showing an error due to RAM or other resource driven error.

However, I did notice that there are errors every few hours that begin with

Source = spid113 "I/O is frozen on database model...." 

a second later with Source = backup "Database backed up....."

a second later with Source = logon "Error 18456 severity 14 State 11"

a second later with source = logon "login failed for user 'domainname\backup'.......

Seeing the I/O leads me to the network which the GIS server logs were also hinting towards with the TCP and closed connection reference. I have reached out to my network guru and he has indicated that they are backing up my servers every 2 hours where we decided this may be playing a role. He has since moved the backup job to nightly. I will continue to monitor the server and SQL logs and will re-post with any updates that I have... hopefully that we fixed this error.

[DEWright_CA]

Ok, so SQL was kicking the connection but because it was locking for a backup.

[BugPie · Accepted solution]

Sharing some good news this morning. I have not seen an error relating to TCP, network or DBMS since I had our network guy make some changes.

What was happening is that when the back up job would run it would freeze the GIS and DBMS servers while it took a snapshot of each server. It would also freeze when the snapshot was reconciled back to one file.

It seems as though this state of "freezing" would cause the connection to the SQL server to stop working. It was not a RAM or resource issue after all.

Beware anyone out there backing up any of their servers during business hours. I now have it set to run once daily at 11pm, about an hour before the GIS server recycles for the day.

[LevinConway]

David,

Thanks for your great analysis of this issue.  This has helped us out with SQL Server 2014.

We were getting the The app domain with specified version id was unloaded due to memory pressure and could not be found error repeatedly, and also many informational messages about app domains being unloaded. 

Upon researching the issue, there turned out to be sufficient RAM allocated to the OS. 

What we discovered is that we were still on the RTM version of SQL Server 2014, and that Microsoft had issued several fixes related to spatial data types and indexes.  Once we upgraded to SQL Server 2014 SP1, cumulative update 1, the memory pressure messages ceased.  The fixes seem to noticibly improve the memory management of spatial data types.