ArcGIS Datastore, published feature layers and a problem with reboot after Windows security updates

[HenriJutila]

We have a ESRI Enterprise 10.9.1 with Server, Portal and Datastore, all installed on same virtual server machine. We have no software modifications at all. Everything is working well exept we have one really annoying issue. Every time we automatically apply security updates for our server's operating system (Windows Server 2019 Datacenter) and the machine reboots, we lose connection to our feature layers that are in the datastore. It seems that the connection between the server and the datastore can't be established properly after the server machine reboot. This happens every time. We can fix the issue by manually rebooting ArcGIS Server -process using taskmanager. We have tried to fix the issue by creating a custom startup process with delayeds starts and by forcing Enterprise components to start up in certain order but that has not helped at all. So, what is going on? Is there a solution for this?  

[Ann-MarieFalkowski1]

We have a very similar issue that has been occurring at 10.8.1 and 10.9.1. Data Store seems to become disconnected from Portal/Server. The Data Store will validate, however, when we try to add a hosted feature layer to a web map we get the message "Error The layer, "name of layer", cannot be added to the map."

Our fix is to run the Configure ArcGIS Server Account utility on the machine hosting ArcGIS Server. When that finishes, we open a command prompt on the Data Store server and run the "configurelicense" command. That fixes the problem without having to reboot or cause any downtime. 

We have not been able to figure out what is causing it or what a permanent fix is. I have a ticket open with Esri. 

[HenriJutila]

Thank you Ann-Marie. I'm sorry to hear that you are also having a problem with Server/Data Store but at the same time it is nice to hear that we are not alone with this weird problem. Also our Data Store will validate, however, feature layers do not work. As long as ESRI or we figure out what is causing the problem we try to use your method for fixing the problem when it occurs in order to avoid downtime. 

-Henri   

[Ann-MarieFalkowski1]

Your post really helped us because we couldn't find what was triggering this issue. Yesterday I looked back at our OS patching and event logs and the patching/restart dates match when the issue is occurring. I will update you when I hear from Esri.

[Ann-MarieFalkowski1]

Hello again! Just wanted to update you on our case with Esri. A Portal for ArcGIS patch was released on 11/14, and was updated on 11/21: https://support.esri.com/en/Products/Enterprise/portal-for-arcgis/portal-for-arcgis/10-9-1#downloads.... Esri wants us to apply this patch. They also had us update all of the Enterprise components (Portal, Server, Data Store) Window Services from "Automatic (Delayed Start)" to "Automatic."

Our next set of patches does not get applied until December so it will be some time before we can know if this fixes the issue. 

[HenriJutila]

Hi! Thank you for this new information. We will apply all patches for 10.9.1, including the one you mentioned. We will also set services to Automatic instead of Automatic (Delayed start). Our OS updates will apply every  third tuesday of the month, so we now just have to wait and see what happens then.

-Henri 

[Dan_Brumm]

FYI, we have been impacted by this same issue for the last two years. We run Enterprise 10.8.1 in HA and if we allow our servers to reboot on their own following OS patching. It will almost always cause an issue with our DS. 

I opened a ticket with ESRI and we worked on this for months but they could never figure it out. Basically because we had a simple work around to get the DS working again when it occured we just kind of gave up because we were spending so much time trying to figure it out. 

Dan 

Dan 

[Ann-MarieFalkowski1]

Hello,

We closed our support case with Esri today. It was determined that this is a defect: 

• BUG-000147597: Upgrading a base ArcGIS Enterprise deployment causes hosted services to fail intermittently after a machine reboot.

We found that the true fix was stopping and starting the ArcGIS Server Windows service. Esri mentioned a possible workaround of setting up a dependency of ArcGIS Server on Portal. We also tried a workaround of increasing start up times on the ArcGIS Server Service, but that did not work.

Our temporary workaround is to have a script stop and start the ArcGIS Server Service each morning. 

[HenriJutila]

Hi,

Thanks for the update. We also increased start up times and also adjusted the start up order of different Enterprise components, but these did not work. Now we are just restarting our ArcGIS server service when needed. Annoying, but what can you do.

-Henri