Help Hosted Feature Layers aren't working anymore.

[JenniferMcFarland]

This issue didn't start untill last night. I have 3 portal layers (hosted feature layers) that are supposed to update every evening. They didn't update last night. I wasn't aware of this issue till the customer reported that their apps were getting an error that said couldn't add layer to the map.   This morning I created a new map image service that works great. The issue alone is with hosted feature layers. All other services that are not hosted feature layers draw just fine.
Here is a rundown of my issues... 1. Hosted feature layers are unable to be added to map. 2. When I go to the rest endpoints for the hosted feature layers I see a list of the layers but when you click on one it goes no further. So it is like they are there but it cant see them. https://geo1.oit.ohio.gov/arcgis/rest/services/Hosted 3. IN any application that uses a hosted feature layer it gets a message that layer can't be added to the map. 4. I can't upload a csv file or anything that I wish to create a hosted feature layer. The error I get states Error While analyzing file. There was an error trying to analyze your file. Please check that file is formatted cortrectly. I have tried this with several different csv files and all produce the same results. 5. I can upload an image. I have no problem doing that. I can also upload using the Add CSV file only option. 6. I validated the datastore and it validated correctly. 7. I checked the federation and it is valid also. 7. I tried adding a hosted feature layer in pro also and this also failed. So this was all working yesterday and it is not working today. They did patch our server last night so I had them roll back the patches and it is still not working. I need to addresss this issue asap and get it working for my end users. 

[ABishop]

Hello Jennifer,

Sounds like something happened with your connection or ability to publish feature layers?  What version of ArcGIS are you using?  ArcMap?  ArcGIS Pro?  Also, which patch was applied on the server?

[Yanlin_Li]

Hi Jennifer,

What does the portal log say about the 3 hosted feature services?   What about the other hosted feature services work except the 3 hosted feature services we are having trouble with? Also, Have you tried to stop the services and start them manually or restart the portal services? 

Yanlin

[ThienTran]

Appears to be an issue related to connectivity between ArcGIS Server and ArcGIS DataStore. Although, you mentioned datastore is validating correctly. Did you validate through Server Manager or Server Admin API?

First thing I would do is restart the Server and DataStore services, validate DataStore again through Server Manager and check logs if there are any errors. Following that, use the blog below to check that the expected processes are running for each Enterprise component in Task Manager:

• Overview of processes that run to support ArcGIS Enterprise components 

[Matthew_Deveney]

I wanted to get this posted for others somewhere - I've had this issue several times over the past three years and the only solution that I have found, after several support tickets with Esri is the answer from @ThienTran above. If an initial restart of each service does not work, then I do it again until it does.

[Jonathan_Marin]

Have you found out what the cause of the issue is though? 

[tigerwoulds]

We experienced this as well after Windows Server patching/updates. Data Store service is running, validates with ArcGIS Server, hosted services are started, but they do not load in Portal's map viewer nor in ArcGIS Pro. 

Running the describe data store utility did not show anything out of the norm. Restarting the Data Store service did not resolve. 

Fix: Reboot ArcGIS Enteprise components in this order:

Stop: Data Store, Server, Portal

Wait a few minutes

Start: Data Store, Server, wait a few minutes, Start Portal, wait a few minutes

This allowed hosted layers to work in Portal now. In ArcGIS Pro 3.1 and 3.2 we had to start with new Projects for the hosted layers to begin working. Existing maps continued to show the same errors, even after re-adding layers/updating sources. 

Also wanted to note: This outage has happened twice before. In those instances we ran the Configure ArcGIS Server account and input the same credentials we had used before, and this also solved the issue. Rebooting windows services fixed it for us in this instance but the configure service account might be something to try if that doesn't work. 

[ViktorsA]

Huge, thanks! You saved me from a big trouble. Solution with shutdown services in right order and then starting them up in exact order worked perfect for me. Hosted services are reachable again now.

[Thomas_Puthusserry]

In one of our test instance (HA) we start and stop the VMs daily (gracefully) - the issue with hosted feature services and hosted 3D scene layers is a common issue. After many attempts, I just stop and start the arcgis service in the gis server and that brings back the hosted feature and 3d scenes without any issue. Initially I followed the steps as above: datastore, server and portal, but it seems the connection issue is just with gis server (at least in our case).