topic Re: ArcGIS Data Store Hosted Feature Layer stop responding in ArcGIS Enterprise Questions

ArcGIS Data Store Hosted Feature Layer stop responding

AndreaV — Fri, 11 Mar 2022 15:15:41 GMT

Hi all,

I have a single server base deployment of ArcGIS Enterprise with a lot of Feature Layer (Hosted) Services.

After a while I notice that this services are no more responding, I get errors like:

"The layer, xxx, cannot be added to the map" in my Web Map Application
or the rest URL of the feature service s"/arcgis/rest/services/Hosted/xxx/FeatureServer" appear as a blank page.

In arcgis server log I found errors similar to:

"Unable to process request. Service Hosted/xxx/MapServer not found"

"Error getting service" for all request to FL Hosted.

I investigated through "describedatastore.bat" utility and all seems to be ok

Information for relational data store ds_vgnupexh
==================================================
Backup location.....................E:\arcgisdatastore\backup\relational
Is backup folder shared.............false
Backup schedule.....................{"schedule-starttime":"00:00:00","schedule-frequency":"Every 0 DAYS"}
Days backup retained................7
Data store status...................Started
member machines.....................MyServer
Maximum connections.................150
Owning system URL...................https://MyServer/arcgis
Portal for ArcGIS URL...............https://MyServer.domain.it/portal
Number of connections...............0 connection(s) to managed database
Data Store mode.....................READWRITE
Is Point-in-time recovery enabled...No
Query optimizer enabled.............Yes

==================================================

and I can validate the relational datastore through ArcGIS Server Manager -> Data Stores -> Validate.

How can I investigate and solve this problem?

thanks in advance

Andrea

Re: ArcGIS Data Store Hosted Feature Layer stop responding

jcarlson — Fri, 11 Mar 2022 15:22:54 GMT

You can try getting into the DataStore log files and see what they say.

We recently had a strange issue with our DataStore. I couldn't quite figure out why it happened, but the old IT standard solution "turn it off and back on again" fixed it.

Re: ArcGIS Data Store Hosted Feature Layer stop responding

AndreaV — Tue, 15 Mar 2022 09:52:34 GMT

Thank you for your reply @jcarlson , I already tried to restart Enterprise services days ago and the whole server too, but the problem remain. Here is the last recent log in \arcgisdatastore\logs\MyServer\server:

<Msg time="2022-03-11T13:54:56,675" type="WARNING" code="110334" source="Data Store" process="6568" thread="21" methodName="" machine="MyServer" user="" elapsed="" requestID="">Validate: Failed to check layers extent.</Msg>
<Msg time="2022-03-11T23:59:59,462" type="INFO" code="110417" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Starting data store machine maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-12T00:00:00,30" type="INFO" code="110217" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Started ArcGIS Data Store configuration store maintenance.</Msg>
<Msg time="2022-03-12T00:00:00,33" type="INFO" code="110218" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Finished ArcGIS Data Store configuration store maintenance.</Msg>
<Msg time="2022-03-12T00:00:00,33" type="INFO" code="110515" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Started relational data store vacuum/analyze maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-12T00:01:46,687" type="WARNING" code="110130" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-03-12T00:02:04,678" type="WARNING" code="110517" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Error during relational data store vacuum/analyze maintenance on 'MyServer'. Failed to execute a SQL script.</Msg>
<Msg time="2022-03-12T00:02:08,534" type="INFO" code="110418" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Successfully completed maintenance on data store machine 'MyServer'.</Msg>
<Msg time="2022-03-12T00:02:22,694" type="WARNING" code="110130" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-03-12T00:02:35,748" type="WARNING" code="110586" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Failed to create index on '{0}'. No valid connection to ArcGIS Data Store configuration store established.
Caused by: Connection to localhost:9876 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.</Msg>
<Msg time="2022-03-12T23:59:59,463" type="INFO" code="110417" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Starting data store machine maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-12T23:59:59,614" type="INFO" code="110217" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Started ArcGIS Data Store configuration store maintenance.</Msg>
<Msg time="2022-03-12T23:59:59,616" type="INFO" code="110218" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Finished ArcGIS Data Store configuration store maintenance.</Msg>
<Msg time="2022-03-12T23:59:59,616" type="INFO" code="110515" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Started relational data store vacuum/analyze maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-13T00:01:46,499" type="WARNING" code="110130" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-03-13T00:04:14,868" type="INFO" code="110516" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Finished relational data store vacuum/analyze maintenance on 'MyServer'.</Msg>
<Msg time="2022-03-13T00:04:14,876" type="INFO" code="110418" source="Data Store" process="5044" thread="1" methodName="" machine="MyServer" user="" elapsed="" requestID="">Successfully completed maintenance on data store machine 'MyServer'.</Msg>
<Msg time="2022-03-14T09:14:06,392" type="WARNING" code="110334" source="Data Store" process="6568" thread="21" methodName="" machine="MyServer" user="" elapsed="" requestID="">Validate: Failed to check layers extent.</Msg>

Re: ArcGIS Data Store Hosted Feature Layer stop responding

ThienTran — Wed, 30 Mar 2022 00:35:31 GMT

Are there any errors in Datastore's postgres logs? (arcgisdatastore\logs\MyServer\database\<postgresql logs>)

You might be seeing errors along the lines of "terminating connection due to administrator command" or unexpected terminations.

I have seen the ArcGIS Data Store has detected an issue with 'db' error occur a few times with the issue being system resource saturation on the ArcGIS Data Store machine. This causes ArcGIS Data Store to crash.

Usually a restart of your ArcGIS Server and ArcGIS Data Store services should bring your hosted services back up again until the next crash, but this might not always be the case depending on what else may have happened.

Are you able to successfully publish new hosted layers?

I would take a look at the logs and monitor your Datastore machine's resource usage.

Re: ArcGIS Data Store Hosted Feature Layer stop responding

AndreaV — Thu, 31 Mar 2022 14:48:59 GMT

Thank you for your reply @ThienTran , I finally could not manage the root of the problem and opted for a more drastic solution. I restored most recent backup of the whole server that let me have all FHLs up and running, then I tried to:

make a fresh back up with Data Store utility,
uninstall Data Store,
remove all folder,
re-install Data Store
cancel wizard configuration
restore from previous backup with Data Store utility

All that seems to solve the problem. Now I have also upgrade to latest version (10.9.1).

Re: ArcGIS Data Store Hosted Feature Layer stop responding

StefanoSchiavini — Thu, 05 May 2022 12:59:34 GMT

Hi,

I have the same issue, maybe someone has found a less drastic solution than reinstall all the Enterprise suite?

I can validate the datastore both from server/manager and https://server.domain.com:6443/arcgis/admin, but can't load FHL and their REST service and I can't publish any new feature hosted layer from ArcGIS PRO.

Obviously I restarted ArcGIS Server and ArcGIS Data Store services and restart also the server without result

This is the arcgisdatastore\logs\<MYSERVER>\database log:

2022-05-04 10:07:03 CEST: [127244]: [1-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [2-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:07:03 CEST: [127244]: [3-1] ERROR: relation "pg_temp_5.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [4-1] STATEMENT: SELECT logfile_data_id FROM pg_temp_5.SDE_logfiles LIMIT 0
2022-05-04 10:07:03 CEST: [127244]: [5-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [6-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:09:33 CEST: [10220]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:09:33 CEST: [126288]: [1-1] FATAL: no pg_hba.conf entry for host "::1", user "dsadmin", database "dsadmindb", SSL off
2022-05-04 10:09:33 CEST: [126288]: [2-1] DETAIL: Client IP address resolved to "LEONARDO2", forward lookup does not match.
2022-05-04 10:09:34 CEST: [18388]: [6-1] LOG: received fast shutdown request
2022-05-04 10:09:34 CEST: [18388]: [7-1] LOG: aborting any active transactions
2022-05-04 10:09:34 CEST: [17120]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 10:09:34 CEST: [18388]: [8-1] LOG: worker process: logical replication launcher (PID 20012) exited with exit code 1
2022-05-04 10:09:34 CEST: [10388]: [1-1] LOG: shutting down
2022-05-04 10:09:35 CEST: [118424]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [125416]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [125932]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [127184]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [18388]: [9-1] LOG: database system is shut down
2022-05-04 10:11:24 CEST: [26080]: [1-1] LOG: database system was shut down at 2022-05-04 10:09:34 CEST
2022-05-04 10:11:25 CEST: [24604]: [5-1] LOG: database system is ready to accept connections
2022-05-04 10:12:17 CEST: [25404]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:14:04 CEST: [40756]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:14:44 CEST: [33280]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:15:30 CEST: [50212]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:48:33 CEST: [36648]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:48:33 CEST: [21336]: [1-1] FATAL: no pg_hba.conf entry for host "::1", user "dsadmin", database "dsadmindb", SSL off
2022-05-04 10:48:33 CEST: [21336]: [2-1] DETAIL: Client IP address resolved to "LEONARDO2", forward lookup does not match.
2022-05-04 10:48:34 CEST: [24604]: [6-1] LOG: received fast shutdown request
2022-05-04 10:48:34 CEST: [24604]: [7-1] LOG: aborting any active transactions
2022-05-04 10:48:34 CEST: [21368]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 10:48:34 CEST: [24604]: [8-1] LOG: worker process: logical replication launcher (PID 22784) exited with exit code 1
2022-05-04 10:48:34 CEST: [24484]: [1-1] LOG: shutting down
2022-05-04 10:48:35 CEST: [30868]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [26992]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [21424]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [26016]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [24604]: [9-1] LOG: database system is shut down
2022-05-04 10:50:21 CEST: [42204]: [1-1] LOG: database system was shut down at 2022-05-04 10:48:34 CEST
2022-05-04 10:50:23 CEST: [127452]: [5-1] LOG: database system is ready to accept connections
2022-05-04 10:58:06 CEST: [107788]: [1-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [2-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:58:06 CEST: [107788]: [3-1] ERROR: relation "pg_temp_5.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [4-1] STATEMENT: SELECT logfile_data_id FROM pg_temp_5.SDE_logfiles LIMIT 0
2022-05-04 10:58:06 CEST: [107788]: [5-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [6-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 11:38:33 CEST: [33312]: [1-1] ERROR: canceling statement due to user request
2022-05-04 11:38:34 CEST: [127452]: [6-1] LOG: received fast shutdown request
2022-05-04 11:38:34 CEST: [127452]: [7-1] LOG: aborting any active transactions
2022-05-04 11:38:34 CEST: [62932]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 11:38:34 CEST: [43024]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 11:38:35 CEST: [127452]: [8-1] LOG: worker process: logical replication launcher (PID 33312) exited with exit code 1
2022-05-04 11:38:38 CEST: [118480]: [1-1] LOG: shutting down
2022-05-04 11:38:43 CEST: [127452]: [9-1] LOG: database system is shut down

thanks in advance

Stefano

Re: ArcGIS Data Store Hosted Feature Layer stop responding

Anonymous User — Tue, 21 Jun 2022 14:28:56 GMT

Has the cause been identified or is this just a failure of the release?

Re: ArcGIS Data Store Hosted Feature Layer stop responding

brumm1dj — Fri, 24 Jun 2022 22:07:13 GMT

FYI, my datastore logs look almost exactly like yours. I end up with very strange behavior with my datastores. It is a CONSTANT battle to keep them running.

Re: ArcGIS Data Store Hosted Feature Layer stop responding

KatherineMichael — Mon, 10 Oct 2022 12:02:53 GMT

We are having this exact problem with similar errors. It started immediately following a Windows patch to the federated Server machine. Rolling the patch back did not fix the issue. Everything validates fine, no necessary ports are blocked. Traditional services on the federated Server work fine, the issue is only with the hosted services. Existing do not load and cannot publish new hosted services. I thought it might be a permissions issue to the configuration file share, but everything looks correct there. I cannot figure out what happened. Will likely open a Support ticket, but checking here to see if anyone found a resolution that didn't involve a reinstall of DataStore?

Re: ArcGIS Data Store Hosted Feature Layer stop responding

BenClark — Wed, 12 Oct 2022 17:52:19 GMT

Adding onto the pile here with my problems. Been experiencing sporadic issues with hosted feature layers for a few months now, but hosted tile/map image services work just fine. As with others, Data Store validates just fine. Rebooting either the Data Store service or the entire application server yields no results. 😵

Re: ArcGIS Data Store Hosted Feature Layer stop responding

AndreaV — Mon, 19 Dec 2022 16:29:23 GMT

Update 15-dec

I analyzed a similar problem on two other installations, one 10.7.1 and one 10.8, and found that it can be solved by re-running the tool: "Configure ArcGIS Server Account" which applies/corrects permissions on the various installation folders. After that, the ArcGIS Server service automatically restarts and Feature Layer Hosted starts working again. I have found that restarting the entire Windows Server sometimes causes the problem to recur and can be solved by the same method. At this point I am investigating a possible problem related to some Windows update...

Re: ArcGIS Data Store Hosted Feature Layer stop responding

MartinKrál — Fri, 10 Mar 2023 12:58:26 GMT

Thank you @AndreaV . Configure ArcGIS Server Account solved our customer's problem with "blank page" hosted feature service today. Good to know.

Re: ArcGIS Data Store Hosted Feature Layer stop responding

Department_of_Communications__ — Fri, 13 Oct 2023 08:42:49 GMT

Hi, @AndreaV , @MartinKrál

I have exactly the same issue .. same logs results etc. Just for clarity ( don't want to panic or break stuff further) could you tell me what steps you took to resolve the above... I don't want to make assumptions. And thanks for your time for posting and responding

Paulg

Re: ArcGIS Data Store Hosted Feature Layer stop responding

MartinKrál — Fri, 13 Oct 2023 09:18:20 GMT

Hi @Department_of_Communications__

What helped in our customer's case (blank page of hosted feature layers from relational datastore) was to run server utility "Configure ArcGIS Server Account" under ArcGIS Server's service user account

https://enterprise.arcgis.com/en/server/11.0/get-started/windows/what-s-included-with-arcgis-server.htm#ESRI_SECTION1_A257E9A10181416BA314EAFF5BE85D98

Regards

Re: ArcGIS Data Store Hosted Feature Layer stop responding

BenClark — Fri, 13 Oct 2023 11:35:24 GMT

I spent ages trying to fix the same issue and in my case it turned out I'd done a server reboot and the Windows services for ArcGIS Server, ArcGIS Data Store, and Portal for ArcGIS hadn't been started in the proper order. This happened multiple times, too--whenever I'd have to reboot the hosting machine hosted services would run fine for weeks or even months, only to eventually and inexplicably crash. Ever since I began following the service boot steps below I've not had hosted services crash once:

Portal for ArcGIS
ArcGIS Data Store
Wait Ten Minutes, then ArcGIS Server

Your issue could indeed be related to some ArcGIS Server config problem, but if that doesn't do the trick you might want to try this as well.

Re: ArcGIS Data Store Hosted Feature Layer stop responding

KatherineMichael — Fri, 13 Oct 2023 11:42:04 GMT

This turned out to be our solution as well.

Re: ArcGIS Data Store Hosted Feature Layer stop responding

CaseyBox_BOPRC — Thu, 09 Nov 2023 09:20:11 GMT

Same here. We have a single machine deployment. I first ran the Configure ArcGIS Server Account utility and when that didn't resolve the issue I stopped, and restarted, the services in the order above, and the issue was resolved.

Re: ArcGIS Data Store Hosted Feature Layer stop responding

KeithOlsen — Mon, 12 Feb 2024 18:46:40 GMT

I'm going to give this reboot order a try; although, I just noticed that ESRI pushed a new ArcGIS Server patch for Hosted Services Reboot. That sounds promising...

Re: ArcGIS Data Store Hosted Feature Layer stop responding

VickyS — Tue, 13 Feb 2024 10:25:48 GMT

Hi @KeithOlsen

The server with our ArcGIS single machine deployment on restarted yesterday following a MS Defender update and we now have this issue. The restart order has not fixed it for us. Have you had any success? I have logged a support call with ESRI UK.

Re: ArcGIS Data Store Hosted Feature Layer stop responding

jonathanharte — Tue, 13 Feb 2024 10:45:19 GMT

I've had similar issues ..... everything comes back up ok but cannot publish new hosted feature content...

The solution @https://community.esri.com/t5/arcgis-enterprise-questions/preferred-order-of-stopping-portal-arcgis-server/td-p/761887 PaulDavidson1 response, works for me with one extra step .. when Portal is back up and settled - restart the Arcgis Server instance .. wait for 5 mins and it all starts publishing again....
hopefully I haven't confused things .. but this is my goto for restarting