ArcGIS Data Store won't validate

[FredSpataro]

Hello, 

I've got a simple all-in-one enterprise portal setup for development/testing but the Relational data store won't validate.  

Thanks for any suggestions.

From the datastore config wizard: 

From the server manager:

From the server rest admin:

From the server manager logs:

SEVERE Sep 14, 2023, 2:47:44 PM The connection property set was missing a required property or the property value was unrecognized. Bad login user[ ] DataValidator.GPServer
WARNING Sep 14, 2023, 2:47:14 PM Validate: null Data Store
WARNING Sep 14, 2023, 2:41:44 PM Validate: null Data Store
WARNING Sep 14, 2023, 2:19:09 PM ArcGIS Data Store is attempting recovery. Data Store
WARNING Sep 14, 2023, 2:18:27 PM ArcGIS Data Store encountered too many problems; however, the standby machine 'PORTAL.XXXXX.COM' will not proceed with failover due to validation check failures. Data Store
WARNING Sep 14, 2023, 2:18:27 PM Validate: Could not connect to server on machine 'PORTAL'. ArcGIS Data Store or ArcGIS Server on that machine may not be running or the machine is not reachable at this time. Data Store
WARNING Sep 14, 2023, 2:18:26 PM Validate: null Data Store
WARNING Sep 14, 2023, 2:18:26 PM ArcGIS Data Store has detected an issue with 'Machine: PORTAL'. Data Store
WARNING Sep 14, 2023, 2:18:18 PM ArcGIS Data Store has detected an issue with 'Machine: PORTAL'. Data Store
WARNING Sep 14, 2023, 2:18:11 PM ArcGIS Data Store has detected an issue with 'Machine: PORTAL'. Data Store
WARNING Sep 14, 2023, 2:15:16 PM ArcGIS Data Store is attempting recovery. Data Store
WARNING Sep 14, 2023, 2:14:26 PM ArcGIS Data Store encountered too many problems; however, the standby machine 'PORTAL.XXXXX.COM' will not proceed with failover due to validation check failures. Data Store
WARNING Sep 14, 2023, 2:14:26 PM Validate: Could not connect to server on machine 'PORTAL'. ArcGIS Data Store or ArcGIS Server on that machine may not be running or the machine is not reachable at this time. Data Store
WARNING Sep 14, 2023, 2:14:24 PM Validate: null Data Store
WARNING Sep 14, 2023, 2:14:24 PM ArcGIS Data Store has detected an issue with 'Machine: PORTAL'. Data Store
WARNING Sep 14, 2023, 2:14:17 PM ArcGIS Data Store has detected an issue with 'Machine: PORTAL'. Data Store
WARNING Sep 14, 2023, 2:14:09 PM ArcGIS Data Store has detected an issue with 'Machine: PORTAL'. Data Store
WARNING Sep 14, 2023, 1:54:18 PM Failed to validate the data store machine 'PORTAL'. 'Server machine 'https://PORTAL:2443/arcgis/datastoreadmin/machines/PORTAL/validate' returned an error. 'The requested resource could not be found in the system. Please check the parameters and try again.'' Admin

[Scott_Tansley]

have you used the describedatastore command line tool:

https://enterprise.arcgis.com/en/portal/latest/administer/windows/data-store-utility-reference.htm#E...

it can give you some useful diagnostic information.

[A_Wyn_Jones]

The message "ArcGIS Data Store encountered too many problems" can sometimes relate to resource contention. Is the spec of the machine sufficient?

Have you tried restarting the ArcGIS Datastore service?

[FredSpataro]

@Scott_Tansley @A_Wyn_Jones thanks for the suggestions:

1. machine is just installed so no load or anything resource hogging at this point.  Basic performance stats from Task Manager: < 10% CPU utilization & < 50% Memory usage

Also yes, tried a datastore service restart.

2. Output from describedatastore.bat ... nothing seems out of the ordinary expect the tile cache validates and it's member machine is PORTAL while the relational datastore's member machine is PORTAL.{domain}.com .... the FQDN has been a recurring issue for me, especially on machine that aren't joined to the domain ie dev machine and DMZ machines ... 

C:\Program Files\ArcGIS\DataStore\tools>describedatastore.bat

Available ArcGIS Data Stores
==================================================
Relational Data Store
Tile Cache Data Store
Spatiotemporal Big Data Store
Graph Store
Object Store

General information of ArcGIS data stores on PORTAL.XXXX.COM
==================================================
Data store release..................11.1.0.42869
Staging location....................C:\ArcGIS\arcgisdatastore\staging
Log location........................C:\ArcGIS\arcgisdatastore\logs
Free disk space.....................153.00GB
Threshold for READONLY mode.........1024MB

Information for relational data store ds_w9mc1edt
==================================================
Backup location.....................C:\ArcGIS\arcgisdatastore\backup\relational
Is backup folder shared.............false
Backup schedule.....................{"schedule-starttime":"00:00:00","schedule-frequency":"Every 4 DAYS"}
Days backup retained................7
Data store status...................Started
SSL enabled.........................true
member machines.....................PORTAL.XXXX.COM
Maximum connections.................150
Owning system URL...................https://portal.xxxx.com/server
Portal for ArcGIS URL...............https://portal.xxxx.com/portal
Number of connections...............0 connection(s) to managed database
Data Store mode.....................READWRITE
Is Point-in-time recovery enabled...No
Query optimizer enabled.............Yes

Information for tile cache data store tcs_ou0nflv3
==================================================
Data location.......................C:\ArcGIS\arcgisdatastore\nosqldata
Data store status...................Started
Backup location.....................C:\ArcGIS\arcgisdatastore\backup\tilecache
Mode................................primaryStandby
member tile cache machines..........PORTAL
Owning system URL...................https://portal.xxxx.com:6443/arcgis/admin
Portal for ArcGIS URL...............https://portal.xxxx.com/portal




Operation completed successfully.

[Scott_Tansley]

I notice you've enabled SSL.  It may make sense to disable it and then revalidate.  I wonder if the portal/hosting server are unable to trust the cert:

https://enterprise.arcgis.com/en/data-store/latest/install/windows/data-store-utility-reference.htm#...

or test this:

https://enterprise.arcgis.com/en/portal/latest/administer/windows/update-ads-ssl-certificate.htm#ESR... 

[Brian_Wilson]

Doing "allowconnection" will make an entry into pg_hba.conf, to allow a different machine to connect. For example if I want to run postgres tools from my desktop. Connections on localhost should always be allowed by default on the hostssl port.

Regarding self-signed certificates, I don't think Postgres cares at all about that. The page you reference says "Similarly, when the hosting server communicates with the data store, or individual machines within the data store communicate with each other, a self-signed SSL certificate is used. This is sufficient for most organizations."

The postgres docs say they run both host and hostssl on the same port and clients negotiate. So if you want to skip hostssl you need to allow host (unencrypted) connections. I don't see an Esri tool for that, I think they have gone all in on hostssl? I mean, I probably don't want my password auth flying over a network without encryption but I'd be okay with localhost connections. The pg_hba.conf file on my datastore even has ssl turn on for localhost, for example

 grep 127.0 pg_hba.conf
hostssl    all             adm_o77v5             127.0.0.1/32            scram-sha-256
hostssl    all     sde        127.0.0.1/32                scram-sha-256
hostssl    db_0eoas    hsu_9vn8j    127.0.0.1/32    scram-sha-256
hostssl    db_0eoas    ins_nb0tr    127.0.0.1/32    scram-sha-256
hostssl    db_0eoas    sde    127.0.0.1/32    scram-sha-256
hostssl    dsadmindb    adm_o77v5    127.0.0.1/32    scram-sha-256
hostssl    dsadmindb    dsadmin    127.0.0.1/32    scram-sha-256

Esri is saying that you may run CA certificates but it's a network policy decision, the database servers won't care.

I am chasing a similar problem as @FredSpataro I think -- my two DataStores are having "issues" and won't work with each other. Same error messages as Fred but separate machines. The primary is unhappy but does not tell me why and it won't switch to the standby because they have a validation problem. Really with the language they are using ("issues", "validation", "relations") I think I need to bring in a therapist.