WebGISDR failing for Full recovery

SzymonPiskula1 · ‎07-05-2018

Hello,

I have a fairly complex 10.6 setup on multiple windows 2016 with all elements being HA/duplicated:

-Portal

-Hosting AGS Site

--Separate Relational DS cluster

--Separate TileCache DS cluster

-Federated AGS Site

Everything works fine on it, stuff gets published, users can deal with it, hosted layers get created, map services work, all good.

Yesterday i have successfully created a full webgisdr backup (including tilecaches). No errors upon creation of the backup all looks good. Today i was going to test if webgisdr tool works/verify my DR procedure. Sadly upon calling the webgisdr tool with --import option the recovery fails with message:

Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.

Nothing has changed in terms of machines in any of the sites/nodes. All machine names are the same, Load Balancer URLs are the same. Yet it feels like the message indicates a mismatch?

Here is the console output when i run the tool:

==================================================
Starting the WebGIS DR utility.
==================================================
The configuration and base backup time in the current Web GIS
-------------------------------------------------------------
Portal: https://portal.<site>/portal
|
|-- Federated Server: https://mapping.<site>/server
|
|-- Hosting Server: https://hosted.<site>/server
| |
| |-- Relational Data Store: https://ec2amaz-<relational-id>.mydomain.local:2443/arcgis
| |
| |-- TileCache Data Store: https://ec2amaz-<tilecache-id>.mydomain.local:2443/arcgis
Unzipping the backup file:
\\<backups>\July-4-2018-11-29-39-AM-EDT-FULL.webgissite
The backup file has been unzipped in 00hr:12min:15sec.
The backup file was created at July 4, 2018 11:29:39 AM EDT.
The configuration and base backup time in the incoming Web GIS
--------------------------------------------------------------
Portal: https://portal.<site>/portal at 7/4/18 11:24 AM
|
|-- Federated Server: https://mapping.<site>/server at 7/4/18 11:24 AM
|
|-- Hosting Server: https://hosted.<site>/server at 7/4/18 11:24 AM
| |
| |-- TileCache Data Store: https://ec2amaz-<tilecache-id>.mydomain.local:2443/arcgis
| |
| |-- Relational Data Store: https://ec2amaz-<relational-id>.mydomain.local:2443/arcgis

Starting the restore process with the WebGIS DR utility.
Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.
Exiting the WebGIS DR utility.

Here is the 'crucial' section of the additional log file in DEBUG mode:

2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 144][route: {s}->https://hosted.<site>:443] can be kept alive indefinitely
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 144][route: {s}->https://hosted.<site>:443][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager is shutting down
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-144: Close connection
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-144: Close connection
2018-07-05 08:36:55 DEBUG [main] org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager shut down
2018-07-05 08:36:55 DEBUG [main] com.esri.arcgis.webgis.util.WebGISUtil - {"incrementalBackupTimeStamp":1530770427265,"backupMode":"FULL","incrementalRestoreTimeStamp":0,"fullBackupTimeStamp":1530789177251,"fullRestoreTimeStamp":0}
2018-07-05 08:36:55 INFO [main] com.esri.arcgis.webgis.storageservice.file.FileStorageService - Unzipping the backup file:
\\<backups>\July-4-2018-11-29-39-AM-EDT-FULL.webgissite
2018-07-05 08:49:10 INFO [main] com.esri.arcgis.webgis.util.WebGISUtil - The backup file has been unzipped in 00hr:12min:15sec.
2018-07-05 08:49:10 INFO [main] com.esri.arcgis.webgis.service.impl.WebGISDRFrontController - The backup file was created at July 4, 2018 11:29:39 AM EDT.
2018-07-05 08:49:10 DEBUG [main] com.esri.arcgis.webgis.service.impl.WebGISDRFrontController - Failed to validate the current Web GIS.
com.esri.arcgis.webgis.WebGISException: Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:818)
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:221)
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.service(WebGISDRFrontController.java:103)
at com.esri.arcgis.webgis.service.impl.WebGISDRManager.c(WebGISDRManager.java:142)
at com.esri.arcgis.webgis.service.impl.WebGISDRManager.importWebGIS(WebGISDRManager.java:125)
at com.esri.arcgis.webgis.client.WebGISDR.main(WebGISDR.java:103)
2018-07-05 08:49:10 DEBUG [main] com.esri.arcgis.webgis.service.impl.WebGISDRFrontController - Deleting the temp directory \\<share>\tempbackups\WebGISSite1530794203646.
2018-07-05 08:49:12 DEBUG [main] com.esri.arcgis.webgis.client.WebGISDR - Exiting the WebGIS DR utility.
com.esri.arcgis.webgis.WebGISException: Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:224)
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.service(WebGISDRFrontController.java:103)
at com.esri.arcgis.webgis.service.impl.WebGISDRManager.c(WebGISDRManager.java:142)
at com.esri.arcgis.webgis.service.impl.WebGISDRManager.importWebGIS(WebGISDRManager.java:125)
at com.esri.arcgis.webgis.client.WebGISDR.main(WebGISDR.java:103)
Caused by: com.esri.arcgis.webgis.WebGISException: Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:818)
at com.esri.arcgis.webgis.service.impl.WebGISDRFrontController.a(WebGISDRFrontController.java:221)
... 4 common frames omitted
2018-07-05 08:49:12 ERROR [main] com.esri.arcgis.webgis.client.WebGISDR - Failed to validate the ArcGIS Data Store for the web GIS. The ArcGIS Data Store in the web GIS backup does not match the ArcGIS Data Store in the current web GIS.

Regards,

Szymon

JonathanQuinn · ‎09-24-2019

Can you describe what you do to keep tabs on it? For example, what actions do you take to make sure it completes? If the machine locks itself, does it fail? Can you comment with your incident number so I can review the notes?

I'm the Product Engineer that is responsible for the DR tool. I'm not doubting there's a problem, (otherwise you wouldn't be posting about it), but I'll be honest, there is likely more to it than it just hangs. The DR tool acts as any other process on the machine that can run unattended.

JeffTimm · ‎09-24-2019

With the the cmd in admin I can run the -import command. It will run a few processes like unzipping the latest backup and it will start restoring...and then it won't do anything. If I go into the cmd prompt and hit enter then it will continue.

I have also started the -import process and left it only to come back in the morning and hit enter for it to complete.

I have scripted it and intermittently it will work. Most often it will not.

#02317110 WebGISDR - Backup and restore is not working properly

They closed the premium support ticket and referred me to professional services.

I have emails from Professional services if you would like them.

JonathanQuinn · ‎09-24-2019

Sounds like a problem with the Quick Edit:

windows - How and why does QuickEdit mode in Command Prompt freeze applications? - Stack Overflow

Why is my command prompt freezing on Windows 10? - Stack Overflow

If you were to select anything in the command prompt when you're running any .bat file or command line command, it'll pause until you hit Enter. This isn't specific to the DR tool.

The first thing I can suggest is make sure that you don't see the "Select" text at to the top left of the command prompt:

That's an indication you're in "QuickEdit" and the DR tool will never continue until you hit Enter. This won't happen if you schedule the DR tool in Task Scheduler.

JeffTimm · ‎09-24-2019

Why is that the first time anyone has said that to me? I will give this a shot. As far as the task scheduler, I will try that again as well. I had other issues with it that were unrelated to this. I will let you know how this turns out.

JonathanQuinn · ‎09-24-2019

It's definitely an obscure problem. It started when Windows Server 2016 came out, since QuickEdit is enabled by default.