Warning Message in the Portal Logs

SteveMaskol · ‎04-24-2019

I recently installed ArcGIS Portal and Data Store, 10.6.1. They were installed on two different servers. In the Portal logs I am seeing the following warning message.

"The database server was found to be stopped. Re-starting it." The warning code is 217060.

The warning message was repeating numerous times throughout the day for several days. I restarted the Portal service and the Data Store service. Initially, that seemed to resolve the problem. The last warning was logged at 11:00 AM yesterday. This morning the warning message again appeared in the logs. Although, not with the same frequency as in days past. So far, I've seen the message three times.

A PostgresSQL error message is also being generated in the Event Viewer at the same time.

"pg_ctl:another server might be running; trying to start the server anyway"

No errors are found in the Event Viewer on the server where the Data Store has been installed. The servers have not been rebooted or stopped.

This is the first time I've installed Portal and the Data Store and I am not a PostgresSQL expert. Has anyone encountered this problem? Any recommendations on how to get it fixed? Thanks!

JonathanQuinn · ‎04-25-2019

The database will start postgres.exe processes on the Portal machine. Do you see those running? Do you see any errors in the pg_logs within the logs directory of Portal?

SteveMaskol · ‎04-25-2019

There are 17 PostgreSQL Server processes currently running on the Portal machine. Is it expected to be that many? In the logs in pg_logs, I am seeing this error several times.

FATAL: no pg_hba.conf entry for host "::1"

The error includes our ArcGIS Server administrator user name and a database.

Although not classified as fatal, there are numerous instances of three log entries where a connection is received, a connection is authorized and finally disconnection. Is it normal behavior to see that many connections and disconnections?

DanielGarcia · ‎05-29-2019

Hello Steve Maskol,

Did you solve your problem? Could you share it with us?

Regards,

Daniel

SteveMaskol · ‎05-29-2019

Hi Daniel,

I found a way to stop the errors. I don't know if it is the correct solution but it stopped the errors. I found this method off of a forum but I don't recall which one. I think it was a PostgreSQL related forum.

On the server where our Portal site is installed there is a file called pg_hba.conf in the folder \arcgisportal\db\. I added an entry under the "IPv6 local connections:" section in the file.

host all all ::1/128 md5

The "FATAL: no pg_hba.conf entry for host "::1" error stopped. I have been meaning to revisit this to make sure this was the best method but have been too involved with getting all of the SSL certificates set up for Server, Portal and Data Store before I federate. The certificates have been a royal pain.

I hope this is helpful.

Steve

JonathanQuinn · ‎05-29-2019

Just to describe what's happening, Portal runs a command to determine whether the database is up. It doesn't pass in a user in that command so the check assumes the identity of whoever is running the Portal service. The error isn't really a problem, just fills up the logs. We're looking into correcting that.

DavidColey · ‎06-07-2019

Hi Jon - I have seen in my ArcGIS Monitor for Enterprise Portal at 10.7 the same error:

'Database found to be stopped restarting it' - and yes it does not appear to be a real error but does fill. While you are at it, could you guys look into these errors that are now appearing in the 1000s:

86 06/07/2019 11:55 AM 06/07/2019 11:46 AM 9000 SEVERE 1 Machine1.BCC.SCGOV.LOCAL 0 : ERROR: operator does not exist: character varying = integer Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts. Position: 447

87 06/07/2019 11:55 AM 06/07/2019 11:46 AM 9000 SEVERE 1 Machine1.BCC.SCGOV.LOCAL Database error has occurred.

and filling page after page of logs. For us, this framework 9000 error began about May 15 and I simply cannot track it down. I do know it's related to PostgreSql db's because of the 'character varying' portion of the error, and of course the fact that the errors are coming from my hosted site....

Here's another one:

12 06/06/2019 11:03 PM 06/06/2019 11:02 PM 9000 SEVERE 1 Machine2.BCC.SCGOV.LOCAL The submitted field 'lasteditor_count' is not available in the layer '0' of the service 'PlatBoundary'.

This last one also occurs by the 1000s but appears to be from OpenData whenever someone attempts to display the chart from the attribute as displayed in the Table tab of the open data site. I have a specific GeoNet post on this here:

Open Data Mapping and Charting Attributes throws ArcGIS Hosted Site Error

None of these appear to be impacting app performance anywhere because no one has complained but it sure would be nice to track these down and apply a fix....

Thanks,

David

JonathanQuinn · ‎06-10-2019

Appears to be a problem with the data in the ArcGIS Data Store, so it'd be good to determine the request that's coming in which generates that error so you can isolate the potentially offending features. You can increase the logging level for Server or look into checking your IIS/reverse proxy logs.

DavidColey · ‎06-10-2019

Thanks Jonathon. Yes I have tried various combinations of logging level increases and trying to match request to my IIS logs. It's not a datastore problem, It's not a hosted feature layer problem. As I said, these did not start until May 15. To me, that suggests a query problem, most likely from a wab widget, or an api, but the requests are too frequent to be of use trying to match up....

If anyone else encounters this:

86 06/07/2019 11:55 AM 06/07/2019 11:46 AM 9000 SEVERE 1 Machine1.BCC.SCGOV.LOCAL 0 : ERROR: operator does not exist: character varying = integer Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts. Position: 447.

it would be great if you could reply to this post.

JonathanQuinn · ‎06-10-2019

Where are those errors showing up? From the post you referenced, the error is when a hosted feature layer/service is accessed or queried, which means they're coming from your hosting server. While you publish a hosted feature layer to Portal, the data is copied to the ArcGIS Data Store and the service is hosted on the Server. Any request to the service is going to the Server, even within an application in the Portal.

Portal knows nothing about the data within an application, it only serves as place where the application is stored. Again, any requests for data are going to the Server where the service is running and, if it's a hosted service, retrieving the data from the ArcGIS Data Store.

You need to troubleshoot this from the Server/ArcGIS Data Store perspective, not Portal. When I open your OpenData site and monitor the network traffic, the following request fails:

https://ags3.scgov.net/server/rest/services/Hosted/LinearStructure/FeatureServer/0/query?f=json&wher...

If you remove outStatistics, it works. I'm not sure why it doesn't like the outStatistics parameter, but those requests may coincide with the errors in your logs.