ArcGIS Data Store Hosted Feature Layer stop responding

[AndreaV]

Hi all,

I have a single server base deployment of ArcGIS Enterprise with a lot of Feature Layer (Hosted) Services.

After a while I notice that this services are no more responding, I get errors like:

"The layer, xxx, cannot be added to the map" in my Web Map Application
or the rest URL of the feature service s"/arcgis/rest/services/Hosted/xxx/FeatureServer" appear as a blank page.

In arcgis server log I found errors similar to:

"Unable to process request. Service Hosted/xxx/MapServer not found"

"Error getting service" for all request to FL Hosted.

I investigated through "describedatastore.bat" utility and all seems to be ok

Information for relational data store ds_vgnupexh
==================================================
Backup location.....................E:\arcgisdatastore\backup\relational
Is backup folder shared.............false
Backup schedule.....................{"schedule-starttime":"00:00:00","schedule-frequency":"Every 0 DAYS"}
Days backup retained................7
Data store status...................Started
member machines.....................MyServer
Maximum connections.................150
Owning system URL...................https://MyServer/arcgis
Portal for ArcGIS URL...............https://MyServer.domain.it/portal
Number of connections...............0 connection(s) to managed database
Data Store mode.....................READWRITE
Is Point-in-time recovery enabled...No
Query optimizer enabled.............Yes

==================================================

and I can validate the relational datastore through ArcGIS Server Manager -> Data Stores -> Validate.

How can I investigate and solve this problem?

thanks in advance

Andrea

[jcarlson]

You can try getting into the DataStore log files and see what they say.

We recently had a strange issue with our DataStore. I couldn't quite figure out why it happened, but the old IT standard solution "turn it off and back on again" fixed it.

[AndreaV]

Thank you for your reply @jcarlson , I already tried to restart Enterprise services days ago and the whole server too, but the problem remain. Here is the last recent log in \arcgisdatastore\logs\MyServer\server:

<Msg time="2022-03-11T13:54:56,675" type="WARNING" code="110334" source="Data Store" process="6568" thread="21" methodName="" machine="MyServer" user="" elapsed="" requestID="">Validate: Failed to check layers extent.</Msg>
<Msg time="2022-03-11T23:59:59,462" type="INFO" code="110417" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Starting data store machine maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-12T00:00:00,30" type="INFO" code="110217" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Started ArcGIS Data Store configuration store maintenance.</Msg>
<Msg time="2022-03-12T00:00:00,33" type="INFO" code="110218" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Finished ArcGIS Data Store configuration store maintenance.</Msg>
<Msg time="2022-03-12T00:00:00,33" type="INFO" code="110515" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Started relational data store vacuum/analyze maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-12T00:01:46,687" type="WARNING" code="110130" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-03-12T00:02:04,678" type="WARNING" code="110517" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Error during relational data store vacuum/analyze maintenance on 'MyServer'. Failed to execute a SQL script.</Msg>
<Msg time="2022-03-12T00:02:08,534" type="INFO" code="110418" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Successfully completed maintenance on data store machine 'MyServer'.</Msg>
<Msg time="2022-03-12T00:02:22,694" type="WARNING" code="110130" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-03-12T00:02:35,748" type="WARNING" code="110586" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Failed to create index on '{0}'. No valid connection to ArcGIS Data Store configuration store established.
Caused by: Connection to localhost:9876 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.</Msg>
<Msg time="2022-03-12T23:59:59,463" type="INFO" code="110417" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Starting data store machine maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-12T23:59:59,614" type="INFO" code="110217" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Started ArcGIS Data Store configuration store maintenance.</Msg>
<Msg time="2022-03-12T23:59:59,616" type="INFO" code="110218" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Finished ArcGIS Data Store configuration store maintenance.</Msg>
<Msg time="2022-03-12T23:59:59,616" type="INFO" code="110515" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Started relational data store vacuum/analyze maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-13T00:01:46,499" type="WARNING" code="110130" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-03-13T00:04:14,868" type="INFO" code="110516" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Finished relational data store vacuum/analyze maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-13T00:04:14,876" type="INFO" code="110418" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Successfully completed maintenance on data store machine 'MyServer'.</Msg>
<Msg time="2022-03-14T09:14:06,392" type="WARNING" code="110334" source="Data Store" process="6568" thread="21" methodName="" machine="MyServer" user="" elapsed="" requestID="">Validate: Failed to check layers extent.</Msg>

[ThienTran]

Are there any errors in Datastore's postgres logs? (arcgisdatastore\logs\MyServer\database\<postgresql logs>)

You might be seeing errors along the lines of "terminating connection due to administrator command" or unexpected terminations. 

I have seen the ArcGIS Data Store has detected an issue with 'db' error occur a few times with the issue being system resource saturation on the ArcGIS Data Store machine. This causes ArcGIS Data Store to crash.

Usually a restart of your ArcGIS Server and ArcGIS Data Store services should bring your hosted services back up again until the next crash, but this might not always be the case depending on what else may have happened.

Are you able to successfully publish new hosted layers?

I would take a look at the logs and monitor your Datastore machine's resource usage.

[AndreaV · Accepted solution]

Thank you for your reply @ThienTran , I finally could not manage the  root of the problem and opted for a more drastic solution. I restored most recent backup of the whole server that let me have all FHLs up and running, then I tried to:

• make a fresh back up with Data Store utility,
• uninstall Data Store,
• remove all folder,
• re-install Data Store
• cancel  wizard configuration
• restore from previous backup with Data Store utility

All that seems to solve the problem. Now I have also upgrade to latest version (10.9.1).

[StefanoSchiavini]

Hi,

I have the same issue, maybe someone has found a less drastic solution than reinstall all the Enterprise suite?

I can validate the datastore both from server/manager and  https://server.domain.com:6443/arcgis/admin, but can't load FHL and their REST service and I can't publish any new feature hosted layer from ArcGIS PRO.

Obviously I restarted ArcGIS Server and ArcGIS Data Store services and restart also the server without result

This is the arcgisdatastore\logs\<MYSERVER>\database log:

2022-05-04 10:07:03 CEST: [127244]: [1-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [2-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:07:03 CEST: [127244]: [3-1] ERROR: relation "pg_temp_5.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [4-1] STATEMENT: SELECT logfile_data_id FROM pg_temp_5.SDE_logfiles LIMIT 0
2022-05-04 10:07:03 CEST: [127244]: [5-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [6-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:09:33 CEST: [10220]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:09:33 CEST: [126288]: [1-1] FATAL: no pg_hba.conf entry for host "::1", user "dsadmin", database "dsadmindb", SSL off
2022-05-04 10:09:33 CEST: [126288]: [2-1] DETAIL: Client IP address resolved to "LEONARDO2", forward lookup does not match.
2022-05-04 10:09:34 CEST: [18388]: [6-1] LOG: received fast shutdown request
2022-05-04 10:09:34 CEST: [18388]: [7-1] LOG: aborting any active transactions
2022-05-04 10:09:34 CEST: [17120]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 10:09:34 CEST: [18388]: [8-1] LOG: worker process: logical replication launcher (PID 20012) exited with exit code 1
2022-05-04 10:09:34 CEST: [10388]: [1-1] LOG: shutting down
2022-05-04 10:09:35 CEST: [118424]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [125416]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [125932]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [127184]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [18388]: [9-1] LOG: database system is shut down
2022-05-04 10:11:24 CEST: [26080]: [1-1] LOG: database system was shut down at 2022-05-04 10:09:34 CEST
2022-05-04 10:11:25 CEST: [24604]: [5-1] LOG: database system is ready to accept connections
2022-05-04 10:12:17 CEST: [25404]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:14:04 CEST: [40756]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:14:44 CEST: [33280]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:15:30 CEST: [50212]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:48:33 CEST: [36648]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:48:33 CEST: [21336]: [1-1] FATAL: no pg_hba.conf entry for host "::1", user "dsadmin", database "dsadmindb", SSL off
2022-05-04 10:48:33 CEST: [21336]: [2-1] DETAIL: Client IP address resolved to "LEONARDO2", forward lookup does not match.
2022-05-04 10:48:34 CEST: [24604]: [6-1] LOG: received fast shutdown request
2022-05-04 10:48:34 CEST: [24604]: [7-1] LOG: aborting any active transactions
2022-05-04 10:48:34 CEST: [21368]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 10:48:34 CEST: [24604]: [8-1] LOG: worker process: logical replication launcher (PID 22784) exited with exit code 1
2022-05-04 10:48:34 CEST: [24484]: [1-1] LOG: shutting down
2022-05-04 10:48:35 CEST: [30868]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [26992]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [21424]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [26016]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [24604]: [9-1] LOG: database system is shut down
2022-05-04 10:50:21 CEST: [42204]: [1-1] LOG: database system was shut down at 2022-05-04 10:48:34 CEST
2022-05-04 10:50:23 CEST: [127452]: [5-1] LOG: database system is ready to accept connections
2022-05-04 10:58:06 CEST: [107788]: [1-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [2-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:58:06 CEST: [107788]: [3-1] ERROR: relation "pg_temp_5.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [4-1] STATEMENT: SELECT logfile_data_id FROM pg_temp_5.SDE_logfiles LIMIT 0
2022-05-04 10:58:06 CEST: [107788]: [5-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [6-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 11:38:33 CEST: [33312]: [1-1] ERROR: canceling statement due to user request
2022-05-04 11:38:34 CEST: [127452]: [6-1] LOG: received fast shutdown request
2022-05-04 11:38:34 CEST: [127452]: [7-1] LOG: aborting any active transactions
2022-05-04 11:38:34 CEST: [62932]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 11:38:34 CEST: [43024]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 11:38:35 CEST: [127452]: [8-1] LOG: worker process: logical replication launcher (PID 33312) exited with exit code 1
2022-05-04 11:38:38 CEST: [118480]: [1-1] LOG: shutting down
2022-05-04 11:38:43 CEST: [127452]: [9-1] LOG: database system is shut down

 thanks in advance

Stefano

[brumm1dj]

FYI, my datastore logs look almost exactly like yours. I end up with very strange behavior with my datastores. It is a CONSTANT battle to keep them running. 

[Anonymous User]

Has the cause been identified or is this just a failure of the release? 

[KatherineMichael]

We are having this exact problem with similar errors.  It started immediately following a Windows patch to the federated Server machine.  Rolling the patch back did not fix the issue.  Everything validates fine, no necessary ports are blocked.  Traditional services on the federated Server work fine, the issue is only with the hosted services.  Existing do not load and cannot publish new hosted services.  I thought it might be a permissions issue to the configuration file share, but everything looks correct there.  I cannot figure out what happened.  Will likely open a Support ticket, but checking here to see if anyone found a resolution that didn't involve a reinstall of DataStore?  

[BenClark]

Adding onto the pile here with my problems. Been experiencing sporadic issues with hosted feature layers for a few months now, but hosted tile/map image services work just fine. As with others, Data Store validates just fine. Rebooting either the Data Store service or the entire application server yields no results. 😵