WebGISDR failing for Full recovery

[SzymonPiskula1]

Hello,

I have a fairly complex 10.6 setup on multiple windows 2016 with all elements being HA/duplicated:

-Portal

-Hosting AGS Site

--Separate Relational DS cluster

--Separate TileCache DS cluster

-Federated AGS Site

Everything works fine on it, stuff gets published, users can deal with it, hosted layers get created, map services work, all good.

Yesterday i have successfully created a full webgisdr backup (including tilecaches). No errors upon creation of the backup all looks good. Today i was going to test if webgisdr tool works/verify my DR procedure. Sadly upon calling the webgisdr tool with --import option the recovery fails with message:

Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.

Nothing has changed in terms of machines in any of the sites/nodes. All machine names are the same, Load Balancer URLs are the same. Yet it feels like the message indicates a mismatch? 

Here is the console output when i run the tool:

==================================================
Starting the WebGIS DR utility.
==================================================

The configuration and base backup time in the current Web GIS
-------------------------------------------------------------
Portal: https://portal.<site>/portal
|
|-- Federated Server: https://mapping.<site>/server
|
|-- Hosting Server: https://hosted.<site>/server
| |
| |-- Relational Data Store: https://ec2amaz-<relational-id>.mydomain.local:2443/arcgis
| |
| |-- TileCache Data Store: https://ec2amaz-<tilecache-id>.mydomain.local:2443/arcgis

Unzipping the backup file:
\\<backups>\July-4-2018-11-29-39-AM-EDT-FULL.webgissite

The backup file has been unzipped in 00hr:12min:15sec.

The backup file was created at July 4, 2018 11:29:39 AM EDT.

The configuration and base backup time in the incoming Web GIS
--------------------------------------------------------------
Portal: https://portal.<site>/portal at 7/4/18 11:24 AM
|
|-- Federated Server: https://mapping.<site>/server at 7/4/18 11:24 AM
|
|-- Hosting Server: https://hosted.<site>/server at 7/4/18 11:24 AM
| |
| |-- TileCache Data Store: https://ec2amaz-<tilecache-id>.mydomain.local:2443/arcgis
| |
| |-- Relational Data Store: https://ec2amaz-<relational-id>.mydomain.local:2443/arcgis

Starting the restore process with the WebGIS DR utility.

Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.

Exiting the WebGIS DR utility.

Here is the 'crucial' section of the additional log file in DEBUG mode:

2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 144][route: {s}->https://hosted.<site>:443] can be kept alive indefinitely
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 144][route: {s}->https://hosted.<site>:443][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager is shutting down
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-144: Close connection
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-144: Close connection
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager shut down
2018-07-05 08:36:55 DEBUG [main] com.esri.arcgis.webgis.util.WebGISUtil - {"incrementalBackupTimeStamp":1530770427265,"backupMode":"FULL","incrementalRestoreTimeStamp":0,"fullBackupTimeStamp":1530789177251,"fullRestoreTimeStamp":0}
2018-07-05 08:36:55 INFO [main] com.esri.arcgis.webgis.storageservice.file.FileStorageService - Unzipping the backup file:
\\<backups>\July-4-2018-11-29-39-AM-EDT-FULL.webgissite
2018-07-05 08:49:10 INFO [main] com.esri.arcgis.webgis.util.WebGISUtil - The backup file has been unzipped in 00hr:12min:15sec.
2018-07-05 08:49:10 INFO [main] com.esri.arcgis.webgis.service.impl.WebGISDRFrontController - The backup file was created at July 4, 2018 11:29:39 AM EDT.
2018-07-05 08:49:10 DEBUG [main] com.esri.arcgis.webgis.service.impl.WebGISDRFrontController - Failed to validate the current Web GIS.
com.esri.arcgis.webgis.WebGISException: Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:818)
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:221)
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.service(WebGISDRFrontController.java:103)
at com.esri.arcgis.webgis.service.impl.WebGISDRManager.c(WebGISDRManager.java:142)
at com.esri.arcgis.webgis.service.impl.WebGISDRManager.importWebGIS(WebGISDRManager.java:125)
at com.esri.arcgis.webgis.client.WebGISDR.main(WebGISDR.java:103)
2018-07-05 08:49:10 DEBUG [main] com.esri.arcgis.webgis.service.impl.WebGISDRFrontController - Deleting the temp directory \\<share>\tempbackups\WebGISSite1530794203646.
2018-07-05 08:49:12 DEBUG [main] com.esri.arcgis.webgis.client.WebGISDR - Exiting the WebGIS DR utility.
com.esri.arcgis.webgis.WebGISException: Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:224)
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.service(WebGISDRFrontController.java:103)
at com.esri.arcgis.webgis.service.impl.WebGISDRManager.c(WebGISDRManager.java:142)
at com.esri.arcgis.webgis.service.impl.WebGISDRManager.importWebGIS(WebGISDRManager.java:125)
at com.esri.arcgis.webgis.client.WebGISDR.main(WebGISDR.java:103)
Caused by: com.esri.arcgis.webgis.WebGISException: Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:818)
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:221)
... 4 common frames omitted
2018-07-05 08:49:12 ERROR [main] com.esri.arcgis.webgis.client.WebGISDR - Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.

Regards,

Szymon

[JonathanQuinn]

There are a couple of bugs related to GeoAnalytics/BDS and the DR tool:

BUG-000113117 The DR tool incorrectly fails during validation of the spatiotemporal data store during the restore

This is fixed at 10.6.1 and resolves the issue Szymon described.

BUG-000112342 A webgisdr incremental restore fails when the spatiotemporal ArcGIS Data Store is registered to the hosting server and under the description.

This will be fixed in the next version.

What version are you on? You shouldn't see the original error if you're at 10.6.1.

[Harald_ØysteinLund1]

Hi, Jonathan

Thanks for your answer. 

We're on th 10.6 version and we are just in the production deployments so we won't be able to upgrade at this moment. 

For now we will stick to snapshot and file server backup.

[JonathanQuinn]

That should work, keep in mind that the PID files within the install directories may need to get cleaned up after you restore the snapshot otherwise that may cause issues for the software:

Ex.

C:\Program Files\ArcGIS\Portal\framework\etc\pids

C:\Program Files\ArcGIS\DataStore\framework\etc\pids

C:\Program Files\ArcGIS\Server\framework\etc\pids

[JeffTimm]

As of 10.7 it still sticks.  Does no good to script because you have to babysit the process.  Also for some reason it stores all of the backups in the C:/arcgisportal/temp folder.  If you are trying to do a nightly back up and restore it really fills up fast. 

[JonathanQuinn]

What do you mean "sticks"? Does the process not complete? The Portal backups are staged in the C:\arcgisportal\temp directory, and should be deleted once the backup is complete. What files are you seeing left over?

[JeffTimm]

The webgisdr team is aware of the bug and are working on it. BUG-000121969  The files getting left over are the unzipped backups.  I am not sure why they aren't getting deleted automatically.

[JonathanQuinn]

Is that the right BUG number? My records indicate that's related to CloudFormation deployments failing:

BUG-000121969 - Highly Available ArcGIS Enterprise deployments installed on Windows Server 2012 R2 intermittently fail to create the CloudFormation stack if the Portal for ArcGIS content store is set to use an Amazon Web Services (AWS) S3 bucket.

Are the folders the ones named tmp-portal-<operation>-<timestamp>?

[JeffTimm]

It doesn't directly apply but that is what professional services attributed my issues to.  If you would like to help I would love to have it. 

[JonathanQuinn]

Have you reached out to Support about this? Have you tried to run it manually within an elevated prompt?

[JeffTimm]

yes I have done both. It works if I babysit the restore as it just hangs if I don't keep tabs on it. This has gone to premium support and then to professional service.  They told me it is a bug and will be fixed in 10.8