nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Cannot restore ArcGIS Enterprise webgisdr backup on standby environment

NicolasGIS

Hello,

I am trying to replicate an ArcGIS Enteprise environment with each component installed on dedicated Windows Server 2019 VM to a standby environment.

Deployment is composed of:

- Portal for ArcGIS 10.9.1

- ArcGIS Server (#AGS1) hosted 10.9.1 with relational and tileCache datastore

- ArcGIS Server (#AGS2) federated 10.9.1

- Relational datastore 10.9.1 on a VM

- TileCache datastore 10.9.1 on a VM

I modified host file on Portal for ArcGIS, AGS1 and AGS2 servers and I am able to successfully initialize the standby ArcGIS Enterprise. I can then create content and access it from my computer with the same modified host file.

But whenever I restore the original webgisdr backup, though webgisdr reported the restore as successful, I noticed that my relational datastore is no longer registered with the hosted ArcGIS Server.

In the database "server" log, I have the following errors:

<Msg time="2022-04-03T21:55:08,108" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:55:44,140" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:55:54,269" type="WARNING" code="110445" source="Data Store" process="5668" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to allow connections to data store machine '[{"machineName":"GISAGSREC01.COMPANY.COM","adminURL":"https://gisagsrec01.company.com:6443/arcgis/admin","synchronize":false,"underMaintenance":false}]'. No valid connection to ArcGIS Data Store configuration store established.
Caused by: FATAL: the database system is starting up</Msg>
<Msg time="2022-04-03T21:55:54,343" type="WARNING" code="110353" source="Data Store" process="5668" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to update license for relational data store '{0}'.</Msg>
<Msg time="2022-04-03T21:56:22,265" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-03T21:56:58,281" type="WARNING" code="110130" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-04-04T00:03:11,285" type="WARNING" code="110353" source="Data Store" process="32" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to update license for relational data store '{0}'. Cannot configure ArcGIS Data Store with this GIS Server site. ArcGIS Server license is invalid. Make sure the license is Advanced or Standard edition, and that it has not expired.</Msg>

It is systematic. Done it 10 times already.

So I decided to create another environement with exactly the same configuration but other URLs. I filled it with some data and applied the same methodology: create a stanby environment with modified hosts files and restore webgisdr: successful.

So I came to the conclusion that my relational datastore is somehow corrupted but it is currently working fine... Any idea what is going ?

Maybe @JonathanQuinn has an idea ?

Many thanks for your help !

Encontre mais publicações marcadas com

Accepted answers

JonathanQuinn

This is a bug:

BUG-000148139 - The relational ArcGIS Data Store should wait until the database is accessible during a restoration before allowing connections from the registered ArcGIS Server Site.

We'll be fixing this at 11.0.

All comments

KThomasson

So I am getting this error for 10.8.x would this patch be available for that version of enterprise as well.?

NicolasGIS

Yes, there is "ArcGIS Data Store 10.9.1 Durability Enhancement Patch":

https://support.esri.com/en-us/patches-updates/2022/arcgis-data-store-10-9-1-durability-enhancement-patch-8057

KThomasson

has there been a patch for this BUG inquiring minds would like to know?

NicolasGIS

FYI@JonathanQuinn, I was able to solve my issue which seems to be related to the disk performance I was restoring the datastore on.

On the drive it was failing, it has as characteristics: 80 MB/s and 100 IO Operations (both read and write)

I tried on a "better" drive with: 300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write)

and the restore worked 🙂

NicolasGIS

Good news the patch is now available: ArcGIS Data Store 10.9.1 Durability Enhancement Patch ..

but it only partially solved my issue of restore unfortunately ☹️

I no longer have the error:

"No valid connection to ArcGIS Data Store configuration store established." is my logs but I still have

"ArcGIS Data Store has detected an issue with 'db'."... and the restore fails.

<Msg time="2022-09-06T17:17:12,981" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:17:35,106" type="WARNING" code="110379" source="Data Store" process="6652" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to restore 'relational' data store. Failed to start relational data store.</Msg>
<Msg time="2022-09-06T17:17:49,13" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:18:27,233" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg>
<Msg time="2022-09-06T17:18:27,233" type="SEVERE" code="110131" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store encountered too many problems. Failover may be invoked if standby is configured.</Msg>
<Msg time="2022-09-06T17:18:28,280" type="SEVERE" code="110453" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Data store machine 'GISSTOREREC01.COMPANY.COM' has failed.</Msg>

The restore seems to be working fine in the logs from what I can tell but afterward, when starting DB, the following is logged:

2022-09-06 17:17:39 CEST: [2280]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:39 CEST: [92]: [4-1] LOG: redo done at 6/2FFB6D18
2022-09-06 17:17:41 CEST: [5556]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:41 CEST: [1832]: [1-1] FATAL: the database system is starting up
2022-09-06 17:17:42 CEST: [92]: [5-1] FATAL: could not write block 0 in file "base/17591/1257285": No such file or directory
2022-09-06 17:17:42 CEST: [92]: [6-1] CONTEXT: writing block 0 of relation base/17591/1257285
2022-09-06 17:17:43 CEST: [4584]: [6-1] LOG: startup process (PID 92) exited with exit code 1
2022-09-06 17:17:43 CEST: [4584]: [7-1] LOG: aborting startup due to startup process failure
2022-09-06 17:17:43 CEST: [4584]: [8-1] LOG: database system is shut down

The funny thing is that on production datastore, there is no "base/17591" folder either so I don't know why does it try to write in this file...

I guess I am good for another loop with support now !

Anyway, @JonathanQuinn you can thank your team for the patch ! It's much appreciated.

NicolasGIS

Hello @JonathanQuinn ,

I am coming back to you regarding the patch for this bug.

I was told it would be available by mid-August, but it is still not the case.

Any information about that on your side ?

Thanks !

NicolasGIS

Thanks for your reply @JonathanQuinn. I "escalated" the bug and contacted our account manager.

Wait and see...

JonathanQuinn

This issue is not targeted for a patch at the moment. If you need this to be resolved quicker, my recommendation is work with your account manager to escalate the bug to be reviewed by Support Technical Leads and product management.

DavidHoy

Thanks – hope we don’t need to wait until July

NicolasGIS

Our case number was "#02956778 - Issue with relational datastore restoration from webgisdr on standby environment". It's been closed yesterday following Jonathan reply.

DavidHoy

@JonathanQuinn do you have a known postgresql size or other indicator when this might happen?

DavidHoy

Is there a support case I can investigate?

NicolasGIS

Yes, I tried: datastore is successfully registered and I can create new hosted feature layers but as you can imagine, my hosted feature layers from the backup are no longer working afterward.

I have been working with tech support for 4 months on this issue... I tried everything I could imagine 😞

DavidHoy

can you try manually unregistering from the Hosting Server before doing the restore?
And then manually re-register after?

NicolasGIS

I am already freaked out thinking about an upgrade if no patch is available...

Upgrading without any parachute: scary !

NicolasGIS

Thanks for your reply @DavidHoy.

Unfortunately, it also occurs when trying to restore solely the relational data store.

Also, the other components are successfully restored by webgisdr as it does not detect that restore of relational datastore failed

DavidHoy

would love to hear there is a workaround - I cant think of one.

Nicholas - you may need to schedule in doing restores of the individual backups in order

NicolasGIS

Many thanks for your quick reply @JonathanQuinn: much appreciated.

Been trying to solve this issue for a while now: happy to read it's been identified.

Any hope for a patch ? I don't sleep very well since I know we cannot restore our production relational datastore 🙂

Thanks !

JonathanQuinn

NicolasGIS

I forgot to precise that relational datastore backup is 13 Go big. Could it be the reason that the database system is still starting ?

Caused by: FATAL: the database system is starting up

And the reason why with small environment, it works ?!

Participe da Conversa

Membros conectados podem postar, seguir atualizações e mais. Novo aqui? Registre uma conta gratuita.

Links Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.