Frequent "The database server was found to be stopped. Re-starting it."

[MichaelSchoelen1]

In our highly available ArcGIS Enterprise, we are seeing the error message below, logged every 5-10 minutes. As a result, we are seeing portal performance issues.

The database server was found to be stopped. Re-starting it.

When investigating the Portal DB framework logs, we found this error message logged at the same frequency:

The process cannot access the file because it is being used by another process

Our question is: what file is the database trying to access? Is there a resolution to this error.

[MichaelSchoelen]

We were able to resolve this issue. It was caused by (what we believe) was a framework-level conflict with a security monitoring agent--in our case an outdated version of Carbon Black Sensor. Updating the sensor software resolved the issue. Removing it entirely worked too, obviously. 

I would recommend strategically removing (simply disabling didn't do the trick) security agents in a test environment to see what makes it tick. I suspect that Carbon Black was also using some form of Postgres as its internal database, causing a conflict.

[MichaelVolz]

Has carbon black software caused any other issues with GIS at your org?  My org is installing this software on all servers to prevent/alert us to malicious attacks, so any feedback on impacts of carbon black would be greatly appreciated.

[MichaelSchoelen]

We haven't seen any additional issues with Carbon Black, and the issues seemed to resolve once we installed the latest version of the sensor. 

[JianLiu]

Hi, 

We have exactly the same issue, and don't have Carbon Black Sensor. Just wonder if anyone else has figured out and would like to share a resolution? We have Data Store and Portal installed on the same machine though. Not sure if that could contribute to the issue...? 

[MichaelSchoelen]

A single-machine deployment shouldn't cause any problems. From what I saw, this sounds like a conflict with a different piece of software. If you are able, I would start systematically removing some of the agents on the machine (assuming that's possible to do), and monitoring the results. 

[AzinSharaf]

We are getting this message every hour. It is a single box and Base deployment. We have Win Defender on the box (which consumes lots of CPU). 

[vipulsoni]

It seems to be a common and recurring problem with most Portal users. also same with our deployment. when fail to publish scene 3d layers , found that there are other issues listed in log.

[DavidColey]

Hi - In my distributed deployment, I too began experiencing the same error: "Database server found to be stopped. Restarting it".  When this was infrequent, not a big deal..... but recently this began to occur with much greater frequent and has now interrupted both data writes and the portal backup routine when the postgres is offline. Obviously.

In our case, we use McAffee.  We found that by removing the ArcGISPortal.exe and the postgres.exe processes from active McAfee inspections (I think its the McAffee Adaptive Threat Protection Service) and adding them to the trust exclusion that has now eliminated this issue.  So similar to the Carbon Black issue identified above by Michael Schoelen‌.  I also told our security team that these processes along with all the other core processes used by server, datastore, and portal should be trusted and not inspected, per all the help docs.  Jonathan Quinn‌ and Michael Volz‌ thought you guys would like to know as well....

[AnibalMmartinez]

Nos esta pasando lo mismo, tenemos McAffee tambien, voy a probar.

Gracias, 

Anibal MArtinez

Telecom Argentina SA