Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS · ‎04-04-2022

Hello,

I am trying to replicate an ArcGIS Enteprise environment with each component installed on dedicated Windows Server 2019 VM to a standby environment.

Deployment is composed of:

- Portal for ArcGIS 10.9.1

- ArcGIS Server (#AGS1) hosted 10.9.1 with relational and tileCache datastore

- ArcGIS Server (#AGS2) federated 10.9.1

- Relational datastore 10.9.1 on a VM

- TileCache datastore 10.9.1 on a VM

I modified host file on Portal for ArcGIS, AGS1 and AGS2 servers and I am able to successfully initialize the standby ArcGIS Enterprise. I can then create content and access it from my computer with the same modified host file.

But whenever I restore the original webgisdr backup, though webgisdr reported the restore as successful, I noticed that my relational datastore is no longer registered with the hosted ArcGIS Server.

In the database "server" log, I have the following errors:

<Msg time="2022-04-03T21:55:08,108" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:55:44,140" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:55:54,269" type="WARNING" code="110445" source="Data Store" process="5668" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to allow connections to data store machine '[{"machineName":"GISAGSREC01.COMPANY.COM","adminURL":"https://gisagsrec01.company.com:6443/arcgis/admin","synchronize":false,"underMaintenance":false}]'. No valid connection to ArcGIS Data Store configuration store established.
Caused by: FATAL: the database system is starting up</Msg>
<Msg time="2022-04-03T21:55:54,343" type="WARNING" code="110353" source="Data Store" process="5668" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to update license for relational data store '{0}'.</Msg>
<Msg time="2022-04-03T21:56:22,265" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:56:58,281" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-04T00:03:11,285" type="WARNING" code="110353" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to update license for relational data store '{0}'. Cannot configure ArcGIS Data Store with this GIS Server site. ArcGIS Server license is invalid. Make sure the license is Advanced or Standard edition, and that it has not expired.</Msg>

It is systematic. Done it 10 times already.

So I decided to create another environement with exactly the same configuration but other URLs. I filled it with some data and applied the same methodology: create a stanby environment with modified hosts files and restore webgisdr: successful.

So I came to the conclusion that my relational datastore is somehow corrupted but it is currently working fine... Any idea what is going ?

Maybe @JonathanQuinn has an idea ?

Many thanks for your help !

NicolasGIS · ‎04-04-2022

I am already freaked out thinking about an upgrade if no patch is available...

Upgrading without any parachute: scary !

DavidHoy · ‎04-05-2022

can you try manually unregistering from the Hosting Server before doing the restore?
And then manually re-register after?

NicolasGIS · ‎04-05-2022

Yes, I tried: datastore is successfully registered and I can create new hosted feature layers but as you can imagine, my hosted feature layers from the backup are no longer working afterward.

I have been working with tech support for 4 months on this issue... I tried everything I could imagine 😞

DavidHoy · ‎04-05-2022

Is there a support case I can investigate?

NicolasGIS · ‎04-05-2022

Our case number was "#02956778 - Issue with relational datastore restoration from webgisdr on standby environment". It's been closed yesterday following Jonathan reply.

DavidHoy · ‎04-05-2022

Thanks – hope we don’t need to wait until July

JonathanQuinn · ‎04-06-2022

This issue is not targeted for a patch at the moment. If you need this to be resolved quicker, my recommendation is work with your account manager to escalate the bug to be reviewed by Support Technical Leads and product management.

NicolasGIS · ‎04-08-2022

Thanks for your reply @JonathanQuinn. I "escalated" the bug and contacted our account manager.

Wait and see...

NicolasGIS · ‎08-31-2022

Hello @JonathanQuinn ,

I am coming back to you regarding the patch for this bug.

I was told it would be available by mid-August, but it is still not the case.

Any information about that on your side ?

Thanks !

NicolasGIS · ‎09-06-2022

Good news the patch is now available: ArcGIS Data Store 10.9.1 Durability Enhancement Patch ..

but it only partially solved my issue of restore unfortunately ☹️

I no longer have the error:

"No valid connection to ArcGIS Data Store configuration store established." is my logs but I still have

"ArcGIS Data Store has detected an issue with 'db'."... and the restore fails.

<Msg time="2022-09-06T17:17:12,981" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:17:35,106" type="WARNING" code="110379" source="Data Store" process="6652" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to restore 'relational' data store. Failed to start relational data store.</Msg>
<Msg time="2022-09-06T17:17:49,13" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:18:27,233" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:18:27,233" type="SEVERE" code="110131" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store encountered too many problems. Failover may be invoked if standby is configured.</Msg>
<Msg time="2022-09-06T17:18:28,280" type="SEVERE" code="110453" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Data store machine 'GISSTOREREC01.COMPANY.COM' has failed.</Msg>

The restore seems to be working fine in the logs from what I can tell but afterward, when starting DB, the following is logged:

2022-09-06 17:17:39 CEST: [2280]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:39 CEST: [92]: [4-1] LOG: redo done at 6/2FFB6D18
2022-09-06 17:17:41 CEST: [5556]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:41 CEST: [1832]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:42 CEST: [92]: [5-1] FATAL: could not write block 0 in file "base/17591/1257285": No such file or directory
2022-09-06 17:17:42 CEST: [92]: [6-1] CONTEXT: writing block 0 of relation base/17591/1257285
2022-09-06 17:17:43 CEST: [4584]: [6-1] LOG: startup process (PID 92) exited with exit code 1
2022-09-06 17:17:43 CEST: [4584]: [7-1] LOG: aborting startup due to startup process failure
2022-09-06 17:17:43 CEST: [4584]: [8-1] LOG: database system is shut down

The funny thing is that on production datastore, there is no "base/17591" folder either so I don't know why does it try to write in this file...

I guess I am good for another loop with support now !

Anyway, @JonathanQuinn you can thank your team for the patch ! It's much appreciated.