topic Cannot restore ArcGIS Enterprise webgisdr backup on standby environment in High Availability and Disaster Recovery Questions

Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Mon, 04 Apr 2022 13:46:34 GMT

Hello,

I am trying to replicate an ArcGIS Enteprise environment with each component installed on dedicated Windows Server 2019 VM to a standby environment.

Deployment is composed of:

- Portal for ArcGIS 10.9.1

- ArcGIS Server (#AGS1) hosted 10.9.1 with relational and tileCache datastore

- ArcGIS Server (#AGS2) federated 10.9.1

- Relational datastore 10.9.1 on a VM

- TileCache datastore 10.9.1 on a VM

I modified host file on Portal for ArcGIS, AGS1 and AGS2 servers and I am able to successfully initialize the standby ArcGIS Enterprise. I can then create content and access it from my computer with the same modified host file.

But whenever I restore the original webgisdr backup, though webgisdr reported the restore as successful, I noticed that my relational datastore is no longer registered with the hosted ArcGIS Server.

In the database "server" log, I have the following errors:

<Msg time="2022-04-03T21:55:08,108" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:55:44,140" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:55:54,269" type="WARNING" code="110445" source="Data Store" process="5668" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to allow connections to data store machine '[{"machineName":"GISAGSREC01.COMPANY.COM","adminURL":"https://gisagsrec01.company.com:6443/arcgis/admin","synchronize":false,"underMaintenance":false}]'. No valid connection to ArcGIS Data Store configuration store established.
Caused by: FATAL: the database system is starting up</Msg>
<Msg time="2022-04-03T21:55:54,343" type="WARNING" code="110353" source="Data Store" process="5668" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to update license for relational data store '{0}'.</Msg>
<Msg time="2022-04-03T21:56:22,265" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:56:58,281" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-04T00:03:11,285" type="WARNING" code="110353" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to update license for relational data store '{0}'. Cannot configure ArcGIS Data Store with this GIS Server site. ArcGIS Server license is invalid. Make sure the license is Advanced or Standard edition, and that it has not expired.</Msg>

It is systematic. Done it 10 times already.

So I decided to create another environement with exactly the same configuration but other URLs. I filled it with some data and applied the same methodology: create a stanby environment with modified hosts files and restore webgisdr: successful.

So I came to the conclusion that my relational datastore is somehow corrupted but it is currently working fine... Any idea what is going ?

Maybe @JonathanQuinn has an idea ?

Many thanks for your help !

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Mon, 04 Apr 2022 15:31:59 GMT

I forgot to precise that relational datastore backup is 13 Go big. Could it be the reason that the database system is still starting ?

Caused by: FATAL: the database system is starting up

And the reason why with small environment, it works ?!

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

JonathanQuinn — Mon, 04 Apr 2022 18:00:34 GMT

This is a bug:

BUG-000148139 - The relational ArcGIS Data Store should wait until the database is accessible during a restoration before allowing connections from the registered ArcGIS Server Site.

We'll be fixing this at 11.0.

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Mon, 04 Apr 2022 18:28:08 GMT

Many thanks for your quick reply @JonathanQuinn: much appreciated.

Been trying to solve this issue for a while now: happy to read it's been identified.

Any hope for a patch ? I don't sleep very well since I know we cannot restore our production relational datastore 🙂

Thanks !

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

DavidHoy — Tue, 05 Apr 2022 01:44:59 GMT

would love to hear there is a workaround - I cant think of one.

Nicholas - you may need to schedule in doing restores of the individual backups in order

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Tue, 05 Apr 2022 04:32:34 GMT

Thanks for your reply @DavidHoy.

Unfortunately, it also occurs when trying to restore solely the relational data store.

Also, the other components are successfully restored by webgisdr as it does not detect that restore of relational datastore failed

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Tue, 05 Apr 2022 05:43:44 GMT

I am already freaked out thinking about an upgrade if no patch is available...

Upgrading without any parachute: scary !

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

DavidHoy — Tue, 05 Apr 2022 07:16:58 GMT

can you try manually unregistering from the Hosting Server before doing the restore?
And then manually re-register after?

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Tue, 05 Apr 2022 07:38:55 GMT

Yes, I tried: datastore is successfully registered and I can create new hosted feature layers but as you can imagine, my hosted feature layers from the backup are no longer working afterward.

I have been working with tech support for 4 months on this issue... I tried everything I could imagine 😞

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

DavidHoy — Tue, 05 Apr 2022 23:52:38 GMT

Is there a support case I can investigate?

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

DavidHoy — Tue, 05 Apr 2022 23:58:31 GMT

@JonathanQuinn do you have a known postgresql size or other indicator when this might happen?

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Wed, 06 Apr 2022 04:34:40 GMT

Our case number was "#02956778 - Issue with relational datastore restoration from webgisdr on standby environment". It's been closed yesterday following Jonathan reply.

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

DavidHoy — Wed, 06 Apr 2022 04:59:57 GMT

Thanks – hope we don’t need to wait until July

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

JonathanQuinn — Wed, 06 Apr 2022 17:28:11 GMT

This issue is not targeted for a patch at the moment. If you need this to be resolved quicker, my recommendation is work with your account manager to escalate the bug to be reviewed by Support Technical Leads and product management.

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Sat, 09 Apr 2022 05:37:17 GMT

Thanks for your reply @JonathanQuinn. I "escalated" the bug and contacted our account manager.

Wait and see...

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Wed, 31 Aug 2022 08:46:37 GMT

Hello @JonathanQuinn ,

I am coming back to you regarding the patch for this bug.

I was told it would be available by mid-August, but it is still not the case.

Any information about that on your side ?

Thanks !

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Tue, 06 Sep 2022 16:15:50 GMT

Good news the patch is now available: ArcGIS Data Store 10.9.1 Durability Enhancement Patch ..

but it only partially solved my issue of restore unfortunately ☹️

I no longer have the error:

"No valid connection to ArcGIS Data Store configuration store established." is my logs but I still have

"ArcGIS Data Store has detected an issue with 'db'."... and the restore fails.

<Msg time="2022-09-06T17:17:12,981" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:17:35,106" type="WARNING" code="110379" source="Data Store" process="6652" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to restore 'relational' data store. Failed to start relational data store.</Msg>
<Msg time="2022-09-06T17:17:49,13" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:18:27,233" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:18:27,233" type="SEVERE" code="110131" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store encountered too many problems. Failover may be invoked if standby is configured.</Msg>
<Msg time="2022-09-06T17:18:28,280" type="SEVERE" code="110453" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Data store machine 'GISSTOREREC01.COMPANY.COM' has failed.</Msg>

The restore seems to be working fine in the logs from what I can tell but afterward, when starting DB, the following is logged:

2022-09-06 17:17:39 CEST: [2280]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:39 CEST: [92]: [4-1] LOG: redo done at 6/2FFB6D18
2022-09-06 17:17:41 CEST: [5556]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:41 CEST: [1832]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:42 CEST: [92]: [5-1] FATAL: could not write block 0 in file "base/17591/1257285": No such file or directory
2022-09-06 17:17:42 CEST: [92]: [6-1] CONTEXT: writing block 0 of relation base/17591/1257285
2022-09-06 17:17:43 CEST: [4584]: [6-1] LOG: startup process (PID 92) exited with exit code 1
2022-09-06 17:17:43 CEST: [4584]: [7-1] LOG: aborting startup due to startup process failure
2022-09-06 17:17:43 CEST: [4584]: [8-1] LOG: database system is shut down

The funny thing is that on production datastore, there is no "base/17591" folder either so I don't know why does it try to write in this file...

I guess I am good for another loop with support now !

Anyway, @JonathanQuinn you can thank your team for the patch ! It's much appreciated.

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Fri, 16 Sep 2022 09:40:27 GMT

FYI@JonathanQuinn, I was able to solve my issue which seems to be related to the disk performance I was restoring the datastore on.

On the drive it was failing, it has as characteristics: 80 MB/s and 100 IO Operations (both read and write)

I tried on a "better" drive with: 300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write)

and the restore worked 🙂

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

KThomasson — Mon, 16 Oct 2023 13:33:28 GMT

has there been a patch for this BUG inquiring minds would like to know?

Re: Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS — Mon, 16 Oct 2023 13:37:45 GMT

Yes, there is "ArcGIS Data Store 10.9.1 Durability Enhancement Patch":

https://support.esri.com/en-us/patches-updates/2022/arcgis-data-store-10-9-1-durability-enhancement-patch-8057