Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

[NicolasGIS]

Hello,

I am trying to replicate an ArcGIS Enteprise environment with each component installed on dedicated Windows Server 2019 VM to a standby environment.

Deployment is composed of:

- Portal for ArcGIS 10.9.1

- ArcGIS Server (#AGS1) hosted 10.9.1 with relational and tileCache datastore

- ArcGIS Server (#AGS2) federated 10.9.1

- Relational datastore 10.9.1 on a VM

- TileCache datastore 10.9.1 on a VM

I modified host file on Portal for ArcGIS, AGS1 and AGS2 servers and I am able to successfully initialize the standby ArcGIS Enterprise. I can then create content and access it from my computer with the same modified host file.

But whenever I restore the original webgisdr backup, though webgisdr reported the restore as successful, I noticed that my relational datastore is no longer registered with the hosted ArcGIS Server.

In the database "server" log, I have the following errors:

<Msg time="2022-04-03T21:55:08,108" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:55:44,140" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:55:54,269" type="WARNING" code="110445" source="Data Store" process="5668" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to allow connections to data store machine '[{"machineName":"GISAGSREC01.COMPANY.COM","adminURL":"https://gisagsrec01.company.com:6443/arcgis/admin","synchronize":false,"underMaintenance":false}]'. No valid connection to ArcGIS Data Store configuration store established.
Caused by: FATAL: the database system is starting up</Msg>
<Msg time="2022-04-03T21:55:54,343" type="WARNING" code="110353" source="Data Store" process="5668" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to update license for relational data store '{0}'.</Msg>
<Msg time="2022-04-03T21:56:22,265" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:56:58,281" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-04T00:03:11,285" type="WARNING" code="110353" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to update license for relational data store '{0}'. Cannot configure ArcGIS Data Store with this GIS Server site. ArcGIS Server license is invalid. Make sure the license is Advanced or Standard edition, and that it has not expired.</Msg>

It is systematic. Done it 10 times already.

So I decided to create another environement with exactly the same configuration but other URLs. I filled it with some data and applied the same methodology: create a stanby environment with modified hosts files and restore webgisdr: successful.

So I came to the conclusion that my relational datastore is somehow corrupted but it is currently working fine... Any idea what is going ?

Maybe @JonathanQuinn has an idea ?

Many thanks for your help !

[NicolasGIS]

I forgot to precise that relational datastore backup is 13 Go big. Could it be the reason that the database system is still starting ? 

Caused by: FATAL: the database system is starting up

And the reason why with small environment, it works ?!

[JonathanQuinn · Accepted solution]

This is a bug:

BUG-000148139 - The relational ArcGIS Data Store should wait until the database is accessible during a restoration before allowing connections from the registered ArcGIS Server Site.

We'll be fixing this at 11.0.

[DavidHoy]

@JonathanQuinn do you have a known postgresql size or other indicator when this might happen?

[KThomasson]

has there been a patch for this BUG inquiring minds would like to know?

[NicolasGIS]

Yes, there is "ArcGIS Data Store 10.9.1 Durability Enhancement Patch":

https://support.esri.com/en-us/patches-updates/2022/arcgis-data-store-10-9-1-durability-enhancement-...

[KThomasson]

So I am getting this error for 10.8.x would this patch be available for that version of enterprise as well.?

[NicolasGIS]

Many thanks for your quick reply @JonathanQuinn: much appreciated.

Been trying to solve this issue for a while now: happy to read it's been identified.

Any hope for a patch ? I don't sleep very well since I know we cannot restore our production relational datastore  🙂

Thanks !

[DavidHoy]

would love to hear there is a workaround - I cant think of one.

Nicholas - you may need to schedule in doing restores of the individual backups in order

[NicolasGIS]

Thanks for your reply @DavidHoy.

Unfortunately, it also occurs when trying to restore solely the relational data store.

Also, the other components are successfully restored by webgisdr as it does not detect that restore of relational datastore failed