Warning Message in the Portal Logs

[SteveMaskol]

I recently installed ArcGIS Portal and Data Store, 10.6.1. They were installed on two different servers. In the Portal logs I am seeing the following warning message.

"The database server was found to be stopped. Re-starting it." The warning code is 217060.

The warning message was repeating numerous times throughout the day for several days. I restarted the Portal service and the Data Store service. Initially, that seemed to resolve the problem. The last warning was logged at 11:00 AM yesterday. This morning the warning message again appeared in the logs. Although, not with the same frequency as in days past. So far, I've seen the message three times. 

A PostgresSQL error message is also being generated in the Event Viewer at the same time.

"pg_ctl:another server might be running; trying to start the server anyway"

No errors are found in the Event Viewer on the server where the Data Store has been installed. The servers have not been rebooted or stopped.  

This is the first time I've installed Portal and the Data Store and I am not a PostgresSQL expert. Has anyone encountered this problem? Any recommendations on how to get it fixed? Thanks!

[DavidColey]

These are server framework errors that appear in ArcGIS Monitor from my server logs.  So yes, I know Portal has nothing to do with this.  In doing some comparisons, I have found that these errors:

463 06/11/2019 7:34 AM 06/11/2019 7:19 AM 9000 SEVERE 1 MACHINE02PW.BCC.SCGOV.LOCAL The submitted field 'primarymaintenance_count' is not available in the layer '0' of the service 'AthleticField'.

are occurring on 'submitted fields' that are either 1) Domained or 2) Not Domained, but contain null values.  This is occurring on every layer in our OpenData site every time a crawler (similar to your request) or a person submits that query (via selecting an attribute to 'chart') So I am now not surprised to see that it is the outStatistics parameter that is failing the request, the question obviously is why. 

This suggests to me that this is a Hub/Open Data error as it is happening to other users per this post, Rachel Micander‌, for one:

Open Data Mapping and Charting Attributes throws ArcGIS Hosted Site Error

and has been occurring for some time and across server releases.

You should know that our host site server content directory is disabled.  

That being said, I cannot track down where the type casting / character varying errors are coming from.  Those errors only began occurring after May 15 after a very in-elegant Windows Server 2016 set of updates....

[JonathanQuinn]

Hm, the only thing I can think of for the The submitted field 'primarymaintenance_count' is not available in the layer '0' of the service 'AthleticField' error is the schema changed and whatever application making requests to the service, (OpenData, for example) is expecting the old schema. I see a primarymaintenance field, but not a primarymaintenance_count field.

If you were to export the data into an FGDB, create a new service, and then run the same query does it work? ex:

https://ags3.scgov.net/server/rest/services/Hosted/NewService/FeatureServer/0/query?f=json&where=1%3... 

[DavidColey]

Thanks for the reply Jon.  The suffix '_count' was never present on any of our field names on any of our layers.  A sample from earlier different layer, many fields:

13            06/11/2019 12:53 AM 06/11/2019 12:47 AM 9000 SEVERE MACHINE2PW.BCC.SCGOV.LOCAL The submitted field 'maplabel_count' is not available in the layer '0' of the service 'CriticalAreaPlan'.

14            06/11/2019 12:53 AM 06/11/2019 12:47 AM 9000 SEVERE MACHINE2PW.BCC.SCGOV.LOCAL The submitted field 'areastatus_count' is not available in the layer '0' of the service 'CriticalAreaPlan'.

15            06/11/2019 12:53 AM 06/11/2019 12:47 AM 9000 SEVERE MACHINE2PW.BCC.SCGOV.LOCAL The submitted field 'areaname_count' is not available in the layer '0' of the service 'CriticalAreaPlan'.

16            06/11/2019 12:53 AM 06/11/2019 12:47 AM 9000SEVERE  MACHINE1PW.BCC.SCGOV.LOCAL The submitted field 'areadesignation_count' is not available in the layer '0' of the service 'CriticalAreaPlan'.

I suspect this is occurring when Open Data is indexing layers, and likely has to do with the outStatistics query param not functioning correctly somehow.  

[DanielGarcia]

Thank's Steve and Jonathan.

[DavidColey]

Hi, so in a related thread - Frequent "The database server was found to be stopped. Re-starting it."  I just laid out what was killing the arcgisportal.exe and postgres.exe processes in our case.  But I also found that for 10.8.1 the exception listed above by Steve Maskol‌ has changed slightly.   Where Steve listed his fix to the 

pg_hba.conf file located in the folder \arcgisportal\db\. I added an entry under the "IPv6 local connections:" section in the file:

host all all ::1/128 md5

the METHOD has changed from md5 to scram-sha-256.  Otherwise everything else is the same.  Sadly though, the log now reads this over and over

2020-11-05 10:24:24 PST: [98028]: FATAL: password authentication failed for user "ourdomainaccount"
2020-11-05 10:24:24 PST: [98028]: DETAIL: Role "ourdomainaccount" does not exist.
Connection matched pg_hba.conf line 12: "host all all ::1/128 scram-sha-256"

I know that the postgres db wants our domain account listed in its datastore but I don't know how to do that.  It just seems that the posgres db wants the account to be local. Jonathan Quinn‌ thought you'd like to know, but for now, other than filling the log, this does not seem to be causing an issue since we added the processes trust I listed in the related thread.