topic Re: Frequent "The database server was found to be stopped. Re-starting it." in ArcGIS Enterprise Questions

Frequent "The database server was found to be stopped. Re-starting it."

MichaelSchoelen1 — Tue, 05 Mar 2019 19:28:21 GMT

In our highly available ArcGIS Enterprise, we are seeing the error message below, logged every 5-10 minutes. As a result, we are seeing portal performance issues.

The database server was found to be stopped. Re-starting it.

When investigating the Portal DB framework logs, we found this error message logged at the same frequency:

The process cannot access the file because it is being used by another process

Our question is: what file is the database trying to access? Is there a resolution to this error.

Re: Frequent "The database server was found to be stopped. Re-starting it."

JonathanQuinn — Mon, 11 Mar 2019 16:20:48 GMT

If you stop the Portal service, do all Portal processes (ArcGISPortal.exe, javaw.exe, java.exe and postgresql.exe), go away? There may be an orphaned process that results in the errors in the db logs. We check to see whether the database is running, and since it can't start due to a potentially orphaned process, you'll see the errors logs in the Portal logs.

Re: Frequent "The database server was found to be stopped. Re-starting it."

MichaelSchoelen1 — Wed, 20 Mar 2019 17:54:05 GMT

It looks like all Portal processes do go away--some javaw processes remain for the Server and Datastore elements.

We've also rebooted the server, and that solves the issue for a few days. Then it suddenly starts again, like some other process is triggering the problem.

Is is possible for Server or Datastore to be conflicting with the Portal processes?

Re: Frequent "The database server was found to be stopped. Re-starting it."

JonathanQuinn — Thu, 21 Mar 2019 00:21:21 GMT

There's no problem with having Server and Data Store on the same machine as Portal.

Are you seeing this on primary or standby? Do you see anything in the Event Viewer?

Re: Frequent "The database server was found to be stopped. Re-starting it."

MichaelSchoelen1 — Thu, 21 Mar 2019 14:39:32 GMT

99% of the time, this message is appearing on the primary machine, but every once in a while, it pop up on the standby.

Event viewer is showing this error message every time the log entry appears in Portaladmin:

The description for Event ID 0 from source PostgreSQL cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
pg_ctl: another server might be running; trying to start server anyway

Would a reinstallation of portal resolve the issue? Or does it seem like the conflict is a different piece of software?

Re: Frequent "The database server was found to be stopped. Re-starting it."

WoosterBe10 — Thu, 25 Apr 2019 16:48:59 GMT

Hi Michael,

I started my own post, "Warning Message in the Portal Logs", before finding yours. Have you found a resolution to this problem? I am encountering the same issue. I restarted the Portal service. The warning messages are not as frequent but they still show up in the log file. Prior to restarting the service, the messages were showing up in the logs dozens of times throughout the day. Now it is only a few times a day.

We have the Portal site installed on one server and the Data Store installed to another server. In the Task Manager for the server where the Portal site is installed, I see PostgresSQL Server listed 15 times under the Processes tab. Do you know if this is normal behavior? No one is using our Portal site. I am in the process of federating but before I do I would like to get this warning message resolved.

Re: Frequent "The database server was found to be stopped. Re-starting it."

SebastianCabrera — Fri, 26 Apr 2019 14:20:56 GMT

Hi all,

In our Portal there is also a frequent logging of this error, every 2, 3 or 4 hours during a day. We upgraded to 10.7, but these errors ocurred even when we were at 10.5.1 and 10.6. We have a dedicated machine for Portal, junst one machine, no HA, and at the time of these errors in the portal logs, in the event viewer two events are logged from PostgreSQL:

pg_ctl: another server might be running; trying to start server anyway

and after this one

pg_ctl: could not start server
Examine the log output.

PostgreSQL is running and listening on port 7654.

the db.log has the same lines as described by Michael.

So, overall, looks like the same behavior, regardless of HA, or other products installed on the same machine, data store listens on 9876.

Someone figured this out?

Re: Frequent "The database server was found to be stopped. Re-starting it."

MichaelSchoelen — Fri, 26 Apr 2019 16:18:44 GMT

We were able to resolve this issue. It was caused by (what we believe) was a framework-level conflict with a security monitoring agent--in our case an outdated version of Carbon Black Sensor. Updating the sensor software resolved the issue. Removing it entirely worked too, obviously.

I would recommend strategically removing (simply disabling didn't do the trick) security agents in a test environment to see what makes it tick. I suspect that Carbon Black was also using some form of Postgres as its internal database, causing a conflict.

Re: Frequent "The database server was found to be stopped. Re-starting it."

MichaelSchoelen — Fri, 26 Apr 2019 16:21:57 GMT

PostgreSQL listed numerous times is normal behavior, assuming that Portal is the one spinning up those instances. It likes to keep individual task separate, then terminate the instance once it is finished.

Re: Frequent "The database server was found to be stopped. Re-starting it."

JianLiu — Thu, 27 Jun 2019 23:30:44 GMT

Hi,

We have exactly the same issue, and don't have Carbon Black Sensor. Just wonder if anyone else has figured out and would like to share a resolution? We have Data Store and Portal installed on the same machine though. Not sure if that could contribute to the issue...?

Re: Frequent "The database server was found to be stopped. Re-starting it."

MichaelVolz — Fri, 28 Jun 2019 00:53:07 GMT

Has carbon black software caused any other issues with GIS at your org? My org is installing this software on all servers to prevent/alert us to malicious attacks, so any feedback on impacts of carbon black would be greatly appreciated.

Re: Frequent "The database server was found to be stopped. Re-starting it."

MichaelSchoelen — Fri, 28 Jun 2019 13:33:05 GMT

We haven't seen any additional issues with Carbon Black, and the issues seemed to resolve once we installed the latest version of the sensor.

Re: Frequent "The database server was found to be stopped. Re-starting it."

MichaelSchoelen — Fri, 28 Jun 2019 13:38:54 GMT

A single-machine deployment shouldn't cause any problems. From what I saw, this sounds like a conflict with a different piece of software. If you are able, I would start systematically removing some of the agents on the machine (assuming that's possible to do), and monitoring the results.

Re: Frequent "The database server was found to be stopped. Re-starting it."

AzinSharaf — Wed, 18 Sep 2019 16:51:17 GMT

We are getting this message every hour. It is a single box and Base deployment. We have Win Defender on the box (which consumes lots of CPU).

Re: Frequent "The database server was found to be stopped. Re-starting it."

vipulsoni — Thu, 03 Oct 2019 05:18:56 GMT

It seems to be a common and recurring problem with most Portal users. also same with our deployment. when fail to publish scene 3d layers , found that there are other issues listed in log.

Re: Frequent "The database server was found to be stopped. Re-starting it."

DavidColey — Thu, 05 Nov 2020 20:14:12 GMT

Hi - In my distributed deployment, I too began experiencing the same error: "Database server found to be stopped. Restarting it". When this was infrequent, not a big deal..... but recently this began to occur with much greater frequent and has now interrupted both data writes and the portal backup routine when the postgres is offline. Obviously.

In our case, we use McAffee. We found that by removing the ArcGISPortal.exe and the postgres.exe processes from active McAfee inspections (I think its the McAffee Adaptive Threat Protection Service) and adding them to the trust exclusion that has now eliminated this issue. So similar to the Carbon Black issue identified above by Michael Schoelen‌. I also told our security team that these processes along with all the other core processes used by server, datastore, and portal should be trusted and not inspected, per all the help docs. Jonathan Quinn‌ and Michael Volz‌ thought you guys would like to know as well....

Re: Frequent "The database server was found to be stopped. Re-starting it."

HillaryBjorstrom — Sun, 08 Aug 2021 15:39:11 GMT

I have the same issue on 10.8.1. Going to try and do an Esri support on Monday.

Re: Frequent "The database server was found to be stopped. Re-starting it."

AnibalMmartinez — Mon, 05 Dec 2022 14:57:46 GMT

Nos esta pasando lo mismo, tenemos McAffee tambien, voy a probar.

Gracias,

Anibal MArtinez

Telecom Argentina SA

Re: Frequent "The database server was found to be stopped. Re-starting it."

HillaryBjorstrom — Mon, 05 Dec 2022 15:02:08 GMT

HERE ARE THE ANTIVIRUS RECOMMENDATIONS FROM ESRI