Hosted Feature Layer can't load

[StefanoSchiavini]

Hi all,

I have a single server base deployment of ArcGIS Enterprise with a of Feature Layer (Hosted) Services that serve a dashboard, the DBMS is PostgreSQL.

The site where the dashboard is published is https://cartorafia.provincia.lodi.it/

After the update of the windows server 2016 I notice that all this services are no more responding. Two weeks ago i've update the SSL certificate.

I've check all the configuration but I can't find any issue.

I can validate the datastore both from server/manager and  https://server.domain.com:6443/arcgis/admin, but can't load FHL and their REST service and I can't publish any new feature hosted layer from ArcGIS PRO.

Obviously I restarted ArcGIS Server and ArcGIS Data Store services and restart also the server without result

This is the arcgisdatastore\logs\<MYSERVER>\database log:

2022-05-04 10:07:03 CEST: [127244]: [1-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [2-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:07:03 CEST: [127244]: [3-1] ERROR: relation "pg_temp_5.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [4-1] STATEMENT: SELECT logfile_data_id FROM pg_temp_5.SDE_logfiles LIMIT 0
2022-05-04 10:07:03 CEST: [127244]: [5-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:07:03 CEST: [127244]: [6-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:09:33 CEST: [10220]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:09:33 CEST: [126288]: [1-1] FATAL: no pg_hba.conf entry for host "::1", user "dsadmin", database "dsadmindb", SSL off
2022-05-04 10:09:33 CEST: [126288]: [2-1] DETAIL: Client IP address resolved to "LEONARDO2", forward lookup does not match.
2022-05-04 10:09:34 CEST: [18388]: [6-1] LOG: received fast shutdown request
2022-05-04 10:09:34 CEST: [18388]: [7-1] LOG: aborting any active transactions
2022-05-04 10:09:34 CEST: [17120]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 10:09:34 CEST: [18388]: [8-1] LOG: worker process: logical replication launcher (PID 20012) exited with exit code 1
2022-05-04 10:09:34 CEST: [10388]: [1-1] LOG: shutting down
2022-05-04 10:09:35 CEST: [118424]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [125416]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [125932]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [127184]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:09:36 CEST: [18388]: [9-1] LOG: database system is shut down
2022-05-04 10:11:24 CEST: [26080]: [1-1] LOG: database system was shut down at 2022-05-04 10:09:34 CEST
2022-05-04 10:11:25 CEST: [24604]: [5-1] LOG: database system is ready to accept connections
2022-05-04 10:12:17 CEST: [25404]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:14:04 CEST: [40756]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:14:44 CEST: [33280]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:15:30 CEST: [50212]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:48:33 CEST: [36648]: [1-1] LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. 

2022-05-04 10:48:33 CEST: [21336]: [1-1] FATAL: no pg_hba.conf entry for host "::1", user "dsadmin", database "dsadmindb", SSL off
2022-05-04 10:48:33 CEST: [21336]: [2-1] DETAIL: Client IP address resolved to "LEONARDO2", forward lookup does not match.
2022-05-04 10:48:34 CEST: [24604]: [6-1] LOG: received fast shutdown request
2022-05-04 10:48:34 CEST: [24604]: [7-1] LOG: aborting any active transactions
2022-05-04 10:48:34 CEST: [21368]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 10:48:34 CEST: [24604]: [8-1] LOG: worker process: logical replication launcher (PID 22784) exited with exit code 1
2022-05-04 10:48:34 CEST: [24484]: [1-1] LOG: shutting down
2022-05-04 10:48:35 CEST: [30868]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [26992]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [21424]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [26016]: [1-1] FATAL: the database system is shutting down
2022-05-04 10:48:36 CEST: [24604]: [9-1] LOG: database system is shut down
2022-05-04 10:50:21 CEST: [42204]: [1-1] LOG: database system was shut down at 2022-05-04 10:48:34 CEST
2022-05-04 10:50:23 CEST: [127452]: [5-1] LOG: database system is ready to accept connections
2022-05-04 10:58:06 CEST: [107788]: [1-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [2-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 10:58:06 CEST: [107788]: [3-1] ERROR: relation "pg_temp_5.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [4-1] STATEMENT: SELECT logfile_data_id FROM pg_temp_5.SDE_logfiles LIMIT 0
2022-05-04 10:58:06 CEST: [107788]: [5-1] ERROR: relation "hsu_vxwl0.sde_logfiles" does not exist at character 29
2022-05-04 10:58:06 CEST: [107788]: [6-1] STATEMENT: SELECT logfile_data_id FROM hsu_vxwl0.SDE_logfiles LIMIT 0
2022-05-04 11:38:33 CEST: [33312]: [1-1] ERROR: canceling statement due to user request
2022-05-04 11:38:34 CEST: [127452]: [6-1] LOG: received fast shutdown request
2022-05-04 11:38:34 CEST: [127452]: [7-1] LOG: aborting any active transactions
2022-05-04 11:38:34 CEST: [62932]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 11:38:34 CEST: [43024]: [1-1] FATAL: terminating connection due to administrator command
2022-05-04 11:38:35 CEST: [127452]: [8-1] LOG: worker process: logical replication launcher (PID 33312) exited with exit code 1
2022-05-04 11:38:38 CEST: [118480]: [1-1] LOG: shutting down
2022-05-04 11:38:43 CEST: [127452]: [9-1] LOG: database system is shut down

Maybe someone has found a less drastic solution than reinstall all the Enterprise suite?

Thanks in advance

Stefano

[Scott_Tansley]

Have you tried using the DescribeDatastore command line tool?  ArcGIS Data Store command utility reference—ArcGIS Enterprise | Documentation for ArcGIS Enterprise

You said you've updated the SSL certs, is that in the web adaptor/arcgis server?  If the latter than you need to run this:

ArcGIS Data Store command utility reference—ArcGIS Enterprise | Documentation for ArcGIS Enterprise

I've also seen issues like this when the the server license has expired and the data store needs to pick up the new license to keep running:

ArcGIS Data Store command utility reference—ArcGIS Enterprise | Documentation for ArcGIS Enterprise

Obviously, it could be many things, but they would be the go to ones if you haven't used them already.

[StefanoSchiavini]

thank you @Scott_Tansley ,

I've already done the first two advice.

Now I try to lunch the updatelicense utility, but the server/manager show that the license never expire.

Stefano 

[Scott_Tansley]

This may be a red-herring, but the owing system URL on the relational datastore is:

https://LEONARDO2/server

on the tile cache it's:

https://leonardo2.localdomain.com:6443/arcgis/admin

I'm not convinced that the relational owning URL is correct and may need reconfiguring?  I typically always use fully qualified domain names, and would have expected that here.  Given that the context 'server' suggests use of a web adaptor, I wonder if  your new SSL cert supports leonardo2?  It may just be the FQDN?  Hard to tell without seeing it, but I'd have anticipated a similar entry to the tile-cache?

[JakeNeedle]

Hi Stefano,

Did you ever get a solution to this issue? We are facing the same situation.

Thanks,

Jake