ArcGIS Data Store Hosted Feature Layer stop responding

[AndreaV]

Hi all,

I have a single server base deployment of ArcGIS Enterprise with a lot of Feature Layer (Hosted) Services.

After a while I notice that this services are no more responding, I get errors like:

"The layer, xxx, cannot be added to the map" in my Web Map Application
or the rest URL of the feature service s"/arcgis/rest/services/Hosted/xxx/FeatureServer" appear as a blank page.

In arcgis server log I found errors similar to:

"Unable to process request. Service Hosted/xxx/MapServer not found"

"Error getting service" for all request to FL Hosted.

I investigated through "describedatastore.bat" utility and all seems to be ok

Information for relational data store ds_vgnupexh
==================================================
Backup location.....................E:\arcgisdatastore\backup\relational
Is backup folder shared.............false
Backup schedule.....................{"schedule-starttime":"00:00:00","schedule-frequency":"Every 0 DAYS"}
Days backup retained................7
Data store status...................Started
member machines.....................MyServer
Maximum connections.................150
Owning system URL...................https://MyServer/arcgis
Portal for ArcGIS URL...............https://MyServer.domain.it/portal
Number of connections...............0 connection(s) to managed database
Data Store mode.....................READWRITE
Is Point-in-time recovery enabled...No
Query optimizer enabled.............Yes

==================================================

and I can validate the relational datastore through ArcGIS Server Manager -> Data Stores -> Validate.

How can I investigate and solve this problem?

thanks in advance

Andrea

[AndreaV]

Update 15-dec

I analyzed a similar problem on two other installations, one 10.7.1 and one 10.8, and found that it can be solved by re-running the tool: "Configure ArcGIS Server Account" which applies/corrects permissions on the various installation folders. After that, the ArcGIS Server service automatically restarts and Feature Layer Hosted starts working again. I have found that restarting the entire Windows Server sometimes causes the problem to recur and can be solved by the same method. At this point I am investigating a possible problem related to some Windows update...

[MartinKrál]

Hi

Thank you @AndreaV . Configure ArcGIS Server Account solved our customer's problem with "blank page" hosted feature service today. Good to know.

[Department_of_Communications__]

Hi,   @AndreaV , @MartinKrál 

I have exactly the same issue .. same logs results etc.  Just for clarity ( don't want to panic or break stuff further)  could you tell me what steps you took to resolve the above...   I don't want to make assumptions.   And thanks for your time for posting and responding 

Paulg

[MartinKrál]

Hi @Department_of_Communications__ 

What helped in our customer's case (blank page of hosted feature layers from relational datastore) was to run server utility "Configure ArcGIS Server Account" under ArcGIS Server's  service user account

https://enterprise.arcgis.com/en/server/11.0/get-started/windows/what-s-included-with-arcgis-server....

Regards

[BenClark]

I spent ages trying to fix the same issue and in my case it turned out I'd done a server reboot and the Windows services for ArcGIS Server, ArcGIS Data Store, and Portal for ArcGIS hadn't been started in the proper order. This happened multiple times, too--whenever I'd have to reboot the hosting machine hosted services would run fine for weeks or even months, only to eventually and inexplicably crash. Ever since I began following the service boot steps below I've not had hosted services crash once:

1. Portal for ArcGIS
2. ArcGIS Data Store
3. Wait Ten Minutes, then ArcGIS Server

Your issue could indeed be related to some ArcGIS Server config problem, but if that doesn't do the trick you might want to try this as well.

[KatherineMichael]

This turned out to be our solution as well.  

[CaseyBox_BOPRC]

Same here. We have a single machine deployment. I first ran the Configure ArcGIS Server Account utility and when that didn't resolve the issue I stopped, and restarted, the services in the order above, and the issue was resolved.

[KeithOlsen]

I'm going to give this reboot order a try; although, I just noticed that ESRI pushed a new ArcGIS Server patch for Hosted Services Reboot. That sounds promising...

[VickyS]

Hi @KeithOlsen

The server with our ArcGIS single machine deployment on restarted yesterday following a MS Defender update and we now have this issue.   The restart order has not fixed it for us. Have you had any success?  I have logged a support call with ESRI UK.

[KeithOlsen]

I changed the reboot order and also installed the arcGIS server Hosted Services Reboot patch. We are using Enterprise 10.9.1. Everything seems to be working this morning, but I couldn't tell you which fixed the issue.