ArcGIS Enterprise services not working since restore of windows backup

[wizgis]

Hi Community,

Here is the scenario,

We are on ArcGIS Enterprise 10.9.1 and wanted to upgrade to ArcGIS Enterprise 11.1

We took following backups 

• WebGIS DR backup of ArcGIS Enterprise
• Windows Backup of the machines hosting ArcGIS Enterprise components

After taking the following backups we launched the Setup.exe of Portal 11.1 and got an error that Portal for ArcGIS 11.1 requires Microsoft Visual C++ 2015-2022 redistributable(x64) files to proceed. 

We installed the redistributable files as mentioned in the message and after that we got the message of restarting the machine which we did and, after restarting machine everything was working fine however, hosted services were not working even though DataStores were validating from Manager & Admin endpoint. On further research we found that we are hitting the following BUG-000147597: ArcGIS Enterprise hosted services may fail after a machine restart.

As there was no clear solution/workaround mentioned in the bug document for 10.9.1 we contacted ESRI support and mentioned about the steps that we have followed so far as well as the BUG that we are hitting. They suggested us to restore the windows backup of the machine and trusting ESRI support we did that (thought we were not sure if that was the right step to do) however, it has caused more issues like

• ArcSoc.exe processes not showing in the task manager. 
• Unable to access server manager through WA URL (Can access manager through port URL)
• REST end point of referenced services are not opening. 

We are not sure how to proceed further like should we restart the machine hosting ArcGIS Enterprise or should we restart the windows services of Portal, Server & DataStore or anything else

Any leads would be appreciated. 

[CodyPatterson]

Hey @wizgis 

A couple things to check first, the ArcGIS Server services doesn't show up on Task Manager, but you can access the server manager through port URL, that's a strange thing to me. Are any of your other servers, Datastore, Portal, Server still possibly in 11.1 post backup? I would do a non-intrusive service restart on all machines, wait around 30 minutes, and do a full server restart of all machines. When I upgraded from 11.1 to 11.2, I had a large amount of issues and a service restart and server restart fixed it in the end.

Hope that helps to some extent!

Cody

[simoxu]

Yes, I agree -  restart all your servers and see how it goes.

System restoration is always the last resort. @wizgis 

Good luck with the upgrade.

[wizgis]

@CodyPatterson Just to make sure that we are on same page, from the following screenshot you can see that we don't see any ArcSoc.exe in the task manager however, we do see other process like java, javaw.exe, rmid.exe etc. 

When accessing the server manager through portURL (/:6443/arcgis/manager) we are able to access the server manager and can see the services published to the server site.

Additionally, we have not yet upgraded to 11.1 because of these issues that we are facing and are still on 10.9.1

We have 2 machine deployment where on one machine we have ArcGIS Server, Portal & DataStore installed and on second machine we have web-adaptors installed for server and portal respectively.

So if I am understanding you correctly I should first restart the windows services of Server, Portal & DataStore and then restart both the machines. 

[simoxu]

For the BUG you mentioned, Esri has a patch for it:
ArcGIS Server Hosted Services Restart Patch (esri.com)

I know it's a bit too late to mention this, because you've rolled back the upgrade using "Windows backup", not sure if it is a so-called snapshot of the machine. 

If your snapshots covers all the ArcGIS Enterprise components, you should be able to restore your 10.9.1 environment... but if the snapshots are not complete, e.g. the portal content directory and server config store were not saved at the same time, you may not be able to restore your system using the snapshots.

If you can't rollback your system using the snapshots, you still have WebGISDR backup, you can reinstall 10.9.1 and restore Enterprise using the backup.

Maybe follow up your current issue with Esri Support, see if they can help you with the system restoration.

[wizgis]

@simoxu It would have been great if ESRI had mentioned that the following patch is applicable for 10.9.1 as well on the following link

As on the following link it mentions only 11.1 hence, I didn't explore it further. This could have been communicated in a better way. 

I am also hazy on the terminology here if Windows Backup and Snapshots are same thing or not however, we have an on-premise deployment and with the backup that our IT has taken and restored all the necessary directories like (Config-store, cache, jobs, output, system directory, portal content directory, etc) are there. 

I hoping to avoid the whole uninstall & reinstall process. I'll follow up with support to see what further can be done.

[RyanUthoff]

I'm confused on what the issue is. You stated that you encountered BUG-000147597. But Esri has already released a patch for it. So have you not installed the patch? Or did you install the patch and still encountered the issue and then resorted to restoring from a backup?

Regardless, if you've already restored from a Windows backup, then I guess it's too late to resolve that issue. At this point, it's not going to hurt to just restart the Windows services of ArcGIS Server and see if it comes back online. Also check the ArcGIS Server logs and see if you can find anything.

As a last resort, you might have to do a WebGISDR restore.

[wizgis · Accepted solution]

UPDATE:

After working with ESRI support, the issue of missing ArcSoc.exe process was resolved by creating a new keycodes file as we had an expired GeoEvent license and restarting the windows service of ArcGIS Server. 

Second issue of not being able to access server manager through WA URL was resolved by re-registering the Web Adaptor with ArcGIS Server as it seemed to have lost the connection after the windows image backup.