ArcGIS Enterprise 11.0: webgisdr restore fails

[NicolasGIS]

Hello,

Since I upgraded from ArcGIS Enteprise 10.9.1 to 11.0, I can successfully create a webgisdr backup but cannot restore it. Whenever I try to restore, it fails for ArcGIS Server with the following error:

Failed to restore the ArcGIS Server:

{
"code": 500,
"messages": ["Import site failed with the following error. The store implementation is not connected to a config store. Please call the connect method before invoking any other method.", "Import operation failed and an attempt to rollback the site to its original state also failed. The site may be functional but it is recommended that you unregister all machines in the site and create a new site based off the configuration store that was backed up at 'C:\\Users\\foo\\AppData\\Local\\Temp\\tmp-ags-site-14599189019367108479128716508001500\\config-store' and the server directories backed up at 'C:\\Users\\foo\\AppData\\Local\\Temp\\tmp-ags-site-14599189019367108479128716508001500\\directories'."],
"status": "error"
}

Was able to reproduce on simple sandbox machine with a base ArcGIS Enteprise deployment on which that workflow used to work at 10.9.1.

Any idea ? Maybe @JonathanQuinn ?

All tests done on Windows Server 2019.

Thanks !

[NicolasGIS]

I did some additional testing and noticed that this issue occurs only when done on an upgraded ArcGIS Enterprise deployment.

• ArcGIS Enterprise base deployment 11.0 -> export webgisdr -> import webgisdr -> successul

• ArcGIS Enterprise base deployment 10.9.1 -> export webgisdr -> import webgisdr -> successul -> Upgrade base deployment to 11.0 -> export webgisdr -> import webgisdr -> failed. 

[ChristopherPawlyszyn]

Were you using a hostname.properties file at 10.9.1, by chance?

Starting to look into this further.

[NicolasGIS]

Thanks @ChristopherPawlyszyn for your reply.

I am deploying my environment using ArcGIS Powershell DSC and it does seem to use hostname.properties file:

https://github.com/Esri/arcgis-powershell-dsc/blob/4a33aabf965e2098699bddc499ce1d05003e992d/Modules/...

Thanks for having a look

[ChristopherPawlyszyn]

@NicolasGIS I deployed a 10.9.1 environment and upgraded to 11.0 with PowerShell DSC and did not reproduce the issue you're describing.

In your scenario for the upgrade, did you use PowerShell DSC as well or perform the process manually?

[NicolasGIS]

Thanks @ChristopherPawlyszyn  for your reply and time.

It's really weird... Following your reply I tried again from scratch with the same result: ArcGIS Server fails to restore.

In ArcGIS Server logs, I have plenty of errors such as:

akka.pattern.AskTimeoutException: Recipient[Actor[akka://akka-rediscovery/user/AkkaAppServerActor#-1696377499]] had already been terminated. Sender[null] sent the message of type "com.esri.arcgis.discovery.admin.rediscovery.util.Message".

But I don't think it is the root cause.

The only difference I see between you and I, is that I use my own fork of ArcGIS DSC because of issue 132:
https://github.com/Esri/arcgis-powershell-dsc/issues/132

I prevent DSC from configuring the hostname:

I had open a BUG regarding this issue:
https://my.esri.com/#/support/bugs/BUG-000124901

As you asked about the hostname.properties file, do you think it could be related ?

Regarding the issue itself, I don't know where to look for. Any idea what I could do to troubleshoot this issue ?

Finally, to answer to your question, the upgrade was done with Powershell DSC.

Thanks !

[ChristopherPawlyszyn]

Interesting, I'm not sure of the hostname.properties file as the cause just something I wanted to try to rule-out. I will try again with that modification to the module and see if I am able to reproduce it.

Is there any evidence of a machine name change (FQDN or IP address) during the upgrade process or following the export/import workflow?

[NicolasGIS]

FYI, @ChristopherPawlyszyn, I tried 2 additional tests:

- Manual install of 10.9.1 -> webgidr export -> webgisdr import -> OK -> Manual upgrade to 11.0 -> webgisdr export -> webgisdr import -> OK

- DSC Install of 10.9.1 -> webgidr export -> webgisdr import -> OK -> Manual upgrade to 11.0 -> webgisdr export -> webgisdr import -> OK

So it looks like the issue only occur when upgrading using DSC. Does it help ?

What could DSC be possibly doing differently than a manual install that would prevent ArcGIS Server from importing the site backup ?!

I will make it crash once again, and search any evidence of a machine name change to answer your question.

Thanks !

[ChristopherPawlyszyn]

I'm curious if you've commented-out the related hostname.properties section in the ArcGIS_ServerUpgrade.psm1 file as well:

https://github.com/Esri/arcgis-powershell-dsc/blob/master/Modules/ArcGIS/DSCResources/ArcGIS_ServerU...

If not, there is an additional indication that the hostname.properties file being added during the upgrade may be at fault, especially in light of your additional testing results.

[NicolasGIS]

Well spotted. I was about to test 😉.

Let me give it another run with this modification and I come back to you !

Thanks !