Hosted "There was an error"

[vijaybadugu]

I am getting " There was an error" when I clicked on Data Tab in item details page. Actually, I uploaded a Zip Shape file on to ArcGIS Enterprise and It created two items one is shape file and other hosted feature layer. 

it was successfully uploaded and if you would like to check after couple days, it gets failed with an error mentioned above. I validated ArcGIS Datastore, federated arcgis server connections, ArcGIS Service account permissions on folders, Disk space, Data store is READ_WRITE Mode and everything looks ok . When I run ArcGIS Server Utility tool , it works for few days and again fails on some random day. How to resolve the issue ?

when i try to upload new shapefile, it does not allow me to upload and logs show as 

"ERROR 001369: Failed to create the service. Failed to execute (PublishServiceDefinition). ERROR 001369: Failed to create the service. Failed to execute (PublishServiceDefinition). Failed to execute (Publish Portal Service). Failed.. Job ID:"

I am using  ArcGIS Enterprise 11.1

Here are datastore logs 

2023-11-21 08:34:18 CST: [24068]: [1-1] ERROR: relation "pg_dist_node" does not exist at character 22
2023-11-21 08:34:18 CST: [24068]: [2-1] STATEMENT: SELECT nodename FROM pg_dist_node WHERE position('<Server>' in nodename) > 0 LIMIT 1
2023-11-21 08:34:19 CST: [24068]: [3-1] ERROR: relation "hsu_jrfzh.sde_logfiles" does not exist at character 29
2023-11-21 08:34:19 CST: [24068]: [4-1] STATEMENT: SELECT logfile_data_id FROM hsu_jrfzh.SDE_logfiles LIMIT 0
2023-11-21 08:34:20 CST: [24068]: [5-1] ERROR: relation "pg_temp_5.sde_logfiles" does not exist at character 29
2023-11-21 08:34:20 CST: [24068]: [6-1] STATEMENT: SELECT logfile_data_id FROM pg_temp_5.SDE_logfiles LIMIT 0
2023-11-21 08:34:22 CST: [24068]: [7-1] ERROR: relation "hsu_jrfzh.sde_logfiles" does not exist at character 29
2023-11-21 08:34:22 CST: [24068]: [8-1] STATEMENT: SELECT logfile_data_id FROM hsu_jrfzh.SDE_logfiles LIMIT 0
2023-11-21 08:37:50 CST: [15904]: [1-1] ERROR: relation "pg_dist_node" does not exist at character 22
2023-11-21 08:37:50 CST: [15904]: [2-1] STATEMENT: SELECT nodename FROM pg_dist_node WHERE position('<Server>' in nodename) > 0 LIMIT 1
2023-11-21 08:37:50 CST: [15904]: [3-1] ERROR: relation "hsu_jrfzh.sde_logfiles" does not exist at character 29
2023-11-21 08:37:50 CST: [15904]: [4-1] STATEMENT: SELECT logfile_data_id FROM hsu_jrfzh.SDE_logfiles LIMIT 0
2023-11-21 08:37:50 CST: [15904]: [5-1] ERROR: relation "pg_temp_5.sde_logfiles" does not exist at character 29
2023-11-21 08:37:50 CST: [15904]: [6-1] STATEMENT: SELECT logfile_data_id FROM pg_temp_5.SDE_logfiles LIMIT 0
2023-11-21 08:37:50 CST: [15904]: [7-1] ERROR: relation "hsu_jrfzh.sde_logfiles" does not exist at character 29
2023-11-21 08:37:50 CST: [15904]: [8-1] STATEMENT: SELECT logfile_data_id FROM hsu_jrfzh.SDE_logfiles LIMIT 0

[Dan_Brumm]

After you uploaded the zip file and it published, could you look at the data tab successfully without getting an error?  I get those errors all the time when I try and publish through the Portal front end. For consistancy we use Pro to publish, which works more often.  This is another super frustrating issue we cant seem to resolve. 

[vijaybadugu]

Publishing from pro as a hosted layer ?

[TonyContreras_Frisco_TX]

I am getting the same errors in the database logs for the data store. When it happens, all hosted feature services fail to load. In my situation, it seems to happen when the standby data store is started (I noticed that the postgresql processes never start on that server when I start the Data Store Windows service). When I stop the standby ds and restart the primary ds, everything works. I will remove and re-add the ds to try and resolve this, then update this thread with results.

[ZeeshanKhan1]

Any result of this test you mentioend?

[TonyContreras_Frisco_TX]

Removing and re-adding the standby did not resolve the issue for me. Once I tried re-registering the standby server, I saw different errors related to not being able to replicate from the primary database, and I noticed that no postgres Server processes would start. I unregistered the standby again and the workaround I came up with was to restart the data store Windows Service every 2 hours via a scheduled task. When looking at the Task Manager on the primary, I saw more than 150 instances of PostgreSQL Server running at the time of these errors. If I restarted the service, the number went down, but then increased over time and at about 2 hours and 15 minutes, it reached over 150 again and the hosted services were returning errors again. I created a Premium Support ticket and they were not able to help or identify a cause. After about 3 weeks, I stopped the scheduled task to try and recreate the issue, but it did not happen again. I added the standby again and this time it replicated without any issue. The connections seemed to be from webhooks, which I do not have enabled and Esri does not have an explanation for why webhooks are connecting to my data store.

[ZeeshanKhan1]

@TonyContreras_Frisco_TX  Thanks for the reply, I still have the same error. I am removing the idle connections from the database twice a day... ESRI support is still trying to figure out and they suggest it may be permissions issues. Let's see how we proceed. will keep you posted.

[vijaybadugu]

I would restart the datastore service every time whenever I get this error. so I decided to publish the layer as copy data option to the server

[ZeeshanKhan1]

Thanks @vijaybadugu , our data is already in the ArcGIS Datastore. For me, this error is not related to the data, rather it is related to the DataStore structure itself.