topic Re: Warning Message in the Portal Logs in ArcGIS Enterprise Questions

Warning Message in the Portal Logs

WoosterBe10 — Wed, 24 Apr 2019 15:50:12 GMT

I recently installed ArcGIS Portal and Data Store, 10.6.1. They were installed on two different servers. In the Portal logs I am seeing the following warning message.

"The database server was found to be stopped. Re-starting it." The warning code is 217060.

The warning message was repeating numerous times throughout the day for several days. I restarted the Portal service and the Data Store service. Initially, that seemed to resolve the problem. The last warning was logged at 11:00 AM yesterday. This morning the warning message again appeared in the logs. Although, not with the same frequency as in days past. So far, I've seen the message three times.

A PostgresSQL error message is also being generated in the Event Viewer at the same time.

"pg_ctl:another server might be running; trying to start the server anyway"

No errors are found in the Event Viewer on the server where the Data Store has been installed. The servers have not been rebooted or stopped.

This is the first time I've installed Portal and the Data Store and I am not a PostgresSQL expert. Has anyone encountered this problem? Any recommendations on how to get it fixed? Thanks!

Re: Warning Message in the Portal Logs

JonathanQuinn — Thu, 25 Apr 2019 17:37:44 GMT

The database will start postgres.exe processes on the Portal machine. Do you see those running? Do you see any errors in the pg_logs within the logs directory of Portal?

Re: Warning Message in the Portal Logs

WoosterBe10 — Thu, 25 Apr 2019 19:08:26 GMT

There are 17 PostgreSQL Server processes currently running on the Portal machine. Is it expected to be that many? In the logs in pg_logs, I am seeing this error several times.

FATAL: no pg_hba.conf entry for host "::1"

The error includes our ArcGIS Server administrator user name and a database.

Although not classified as fatal, there are numerous instances of three log entries where a connection is received, a connection is authorized and finally disconnection. Is it normal behavior to see that many connections and disconnections?

Re: Warning Message in the Portal Logs

DanielGarcia — Wed, 29 May 2019 08:03:09 GMT

Hello Steve Maskol,

Did you solve your problem? Could you share it with us?

Regards,

Daniel

Re: Warning Message in the Portal Logs

WoosterBe10 — Wed, 29 May 2019 13:29:41 GMT

Hi Daniel,

I found a way to stop the errors. I don't know if it is the correct solution but it stopped the errors. I found this method off of a forum but I don't recall which one. I think it was a PostgreSQL related forum.

On the server where our Portal site is installed there is a file called pg_hba.conf in the folder \arcgisportal\db\. I added an entry under the "IPv6 local connections:" section in the file.

host all all ::1/128 md5

The "FATAL: no pg_hba.conf entry for host "::1" error stopped. I have been meaning to revisit this to make sure this was the best method but have been too involved with getting all of the SSL certificates set up for Server, Portal and Data Store before I federate. The certificates have been a royal pain.

I hope this is helpful.

Steve

Re: Warning Message in the Portal Logs

JonathanQuinn — Wed, 29 May 2019 17:30:46 GMT

Just to describe what's happening, Portal runs a command to determine whether the database is up. It doesn't pass in a user in that command so the check assumes the identity of whoever is running the Portal service. The error isn't really a problem, just fills up the logs. We're looking into correcting that.

Re: Warning Message in the Portal Logs

DanielGarcia — Wed, 05 Jun 2019 15:04:29 GMT

Thank's Steve and Jonathan.

Re: Warning Message in the Portal Logs

DavidColey — Fri, 07 Jun 2019 20:18:13 GMT

Hi Jon - I have seen in my ArcGIS Monitor for Enterprise Portal at 10.7 the same error:

'Database found to be stopped restarting it' - and yes it does not appear to be a real error but does fill. While you are at it, could you guys look into these errors that are now appearing in the 1000s:

86 06/07/2019 11:55 AM 06/07/2019 11:46 AM 9000 SEVERE 1 Machine1.BCC.SCGOV.LOCAL 0 : ERROR: operator does not exist: character varying = integer Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts. Position: 447

87 06/07/2019 11:55 AM 06/07/2019 11:46 AM 9000 SEVERE 1 Machine1.BCC.SCGOV.LOCAL Database error has occurred.

and filling page after page of logs. For us, this framework 9000 error began about May 15 and I simply cannot track it down. I do know it's related to PostgreSql db's because of the 'character varying' portion of the error, and of course the fact that the errors are coming from my hosted site....

Here's another one:

12 06/06/2019 11:03 PM 06/06/2019 11:02 PM 9000 SEVERE 1 Machine2.BCC.SCGOV.LOCAL The submitted field 'lasteditor_count' is not available in the layer '0' of the service 'PlatBoundary'.

This last one also occurs by the 1000s but appears to be from OpenData whenever someone attempts to display the chart from the attribute as displayed in the Table tab of the open data site. I have a specific GeoNet post on this here:

Open Data Mapping and Charting Attributes throws ArcGIS Hosted Site Error

None of these appear to be impacting app performance anywhere because no one has complained but it sure would be nice to track these down and apply a fix....

Thanks,

David

Re: Warning Message in the Portal Logs

JonathanQuinn — Mon, 10 Jun 2019 18:59:21 GMT

Appears to be a problem with the data in the ArcGIS Data Store, so it'd be good to determine the request that's coming in which generates that error so you can isolate the potentially offending features. You can increase the logging level for Server or look into checking your IIS/reverse proxy logs.

Re: Warning Message in the Portal Logs

DavidColey — Mon, 10 Jun 2019 21:28:58 GMT

Thanks Jonathon. Yes I have tried various combinations of logging level increases and trying to match request to my IIS logs. It's not a datastore problem, It's not a hosted feature layer problem. As I said, these did not start until May 15. To me, that suggests a query problem, most likely from a wab widget, or an api, but the requests are too frequent to be of use trying to match up....

If anyone else encounters this:

it would be great if you could reply to this post.

Re: Warning Message in the Portal Logs

JonathanQuinn — Mon, 10 Jun 2019 21:51:55 GMT

Where are those errors showing up? From the post you referenced, the error is when a hosted feature layer/service is accessed or queried, which means they're coming from your hosting server. While you publish a hosted feature layer to Portal, the data is copied to the ArcGIS Data Store and the service is hosted on the Server. Any request to the service is going to the Server, even within an application in the Portal.

Portal knows nothing about the data within an application, it only serves as place where the application is stored. Again, any requests for data are going to the Server where the service is running and, if it's a hosted service, retrieving the data from the ArcGIS Data Store.

You need to troubleshoot this from the Server/ArcGIS Data Store perspective, not Portal. When I open your OpenData site and monitor the network traffic, the following request fails:

https://ags3.scgov.net/server/rest/services/Hosted/LinearStructure/FeatureServer/0/query?f=json&where=1%3D1&outStatistics=[{%22statisticType%22:%22count%22,%22onStatisticField%22:%22objectid%22,%22outStatisticFieldName%22:%22objectIdCount%22}]

If you remove outStatistics, it works. I'm not sure why it doesn't like the outStatistics parameter, but those requests may coincide with the errors in your logs.

Re: Warning Message in the Portal Logs

DavidColey — Tue, 11 Jun 2019 14:07:25 GMT

These are server framework errors that appear in ArcGIS Monitor from my server logs. So yes, I know Portal has nothing to do with this. In doing some comparisons, I have found that these errors:

463 06/11/2019 7:34 AM 06/11/2019 7:19 AM 9000 SEVERE 1 MACHINE02PW.BCC.SCGOV.LOCAL The submitted field 'primarymaintenance_count' is not available in the layer '0' of the service 'AthleticField'.

are occurring on 'submitted fields' that are either 1) Domained or 2) Not Domained, but contain null values. This is occurring on every layer in our OpenData site every time a crawler (similar to your request) or a person submits that query (via selecting an attribute to 'chart') So I am now not surprised to see that it is the outStatistics parameter that is failing the request, the question obviously is why.

This suggests to me that this is a Hub/Open Data error as it is happening to other users per this post, Rachel Micander‌, for one:

Open Data Mapping and Charting Attributes throws ArcGIS Hosted Site Error

and has been occurring for some time and across server releases.

You should know that our host site server content directory is disabled.

That being said, I cannot track down where the type casting / character varying errors are coming from. Those errors only began occurring after May 15 after a very in-elegant Windows Server 2016 set of updates....

Re: Warning Message in the Portal Logs

JonathanQuinn — Tue, 11 Jun 2019 18:25:13 GMT

Hm, the only thing I can think of for the The submitted field 'primarymaintenance_count' is not available in the layer '0' of the service 'AthleticField' error is the schema changed and whatever application making requests to the service, (OpenData, for example) is expecting the old schema. I see a primarymaintenance field, but not a primarymaintenance_count field.

If you were to export the data into an FGDB, create a new service, and then run the same query does it work? ex:

https://ags3.scgov.net/server/rest/services/Hosted/NewService/FeatureServer/0/query?f=json&where=1%3D1&outStatistics=[{%22statisticType%22:%22count%22,%22onStatisticField%22:%22objectid%22,%22outStatisticFieldName%22:%22objectIdCount%22}]

Re: Warning Message in the Portal Logs

DavidColey — Tue, 11 Jun 2019 20:17:11 GMT

Thanks for the reply Jon. The suffix '_count' was never present on any of our field names on any of our layers. A sample from earlier different layer, many fields:

13 06/11/2019 12:53 AM 06/11/2019 12:47 AM 9000 SEVERE MACHINE2PW.BCC.SCGOV.LOCAL The submitted field 'maplabel_count' is not available in the layer '0' of the service 'CriticalAreaPlan'.

14 06/11/2019 12:53 AM 06/11/2019 12:47 AM 9000 SEVERE MACHINE2PW.BCC.SCGOV.LOCAL The submitted field 'areastatus_count' is not available in the layer '0' of the service 'CriticalAreaPlan'.

15 06/11/2019 12:53 AM 06/11/2019 12:47 AM 9000 SEVERE MACHINE2PW.BCC.SCGOV.LOCAL The submitted field 'areaname_count' is not available in the layer '0' of the service 'CriticalAreaPlan'.

16 06/11/2019 12:53 AM 06/11/2019 12:47 AM 9000SEVERE MACHINE1PW.BCC.SCGOV.LOCAL The submitted field 'areadesignation_count' is not available in the layer '0' of the service 'CriticalAreaPlan'.

I suspect this is occurring when Open Data is indexing layers, and likely has to do with the outStatistics query param not functioning correctly somehow.

Re: Warning Message in the Portal Logs

DavidColey — Thu, 05 Nov 2020 20:25:59 GMT

Hi, so in a related thread - Frequent "The database server was found to be stopped. Re-starting it." I just laid out what was killing the arcgisportal.exe and postgres.exe processes in our case. But I also found that for 10.8.1 the exception listed above by Steve Maskol‌ has changed slightly. Where Steve listed his fix to the

pg_hba.conf file located in the folder \arcgisportal\db\. I added an entry under the "IPv6 local connections:" section in the file:

host all all ::1/128 md5

the METHOD has changed from md5 to scram-sha-256. Otherwise everything else is the same. Sadly though, the log now reads this over and over

2020-11-05 10:24:24 PST: [98028]: FATAL: password authentication failed for user "ourdomainaccount"
2020-11-05 10:24:24 PST: [98028]: DETAIL: Role "ourdomainaccount" does not exist.
Connection matched pg_hba.conf line 12: "host all all ::1/128 scram-sha-256"

I know that the postgres db wants our domain account listed in its datastore but I don't know how to do that. It just seems that the posgres db wants the account to be local. Jonathan Quinn‌ thought you'd like to know, but for now, other than filling the log, this does not seem to be causing an issue since we added the processes trust I listed in the related thread.