11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

[NicolasGIS]

Hello,

I am testing the replication of an Enterprise 11.5 deployment on Windows Server 2022 with 3 datastores (tileCache, relational, and objectStore).

I set up a standby environment on a different set of VMs and restore a webgisdr backup. At first sight, everything is in order and all the components are properly restored and connected one another.

But when having a closer a look at the objectStore, I realized that is now pointing to the original objectStore.

You can notice it:

- from the ArcGIS Server manager when validating the objectStore: the _proxyUrl parameter is:

https://myoriginaldatastoreVM.company.com:2443/arcgis/datastoreadmin/machines/MYORIGINGALDATASTOREVM...  

- from the ArcGIS Server admin interface when checking the machines of the cloud store "objectStore":

https://myags.company.com:6443/arcgis/admin/data/items/cloudStores/AGSDataStore_objectstore_oz_aev7q... 

it is the name of the original datastore VM which is written (ie: "myoriginaldatastore").

Also, if you now check the replicatedDataStore VM, and browse:

https://localhost:2443/arcgis/datastore/

You will see that it is no longer "attached" on the replicated deployment.

I am confused: is it a BUG or a feature ? Did I miss anything @JonathanQuinn maybe ?

Thanks,

/cc FYI @PhilippeVDV @CedricDespierreCorporon 

[NicolasGIS]

And the worst is that it also happens if your webgidr does not include object store (by setting "INCLUDE_GRAPH_STORE_DATA" to false) !
So the only working workflow to fully replicate a 11.5 ArcGIS Enterprise deployment that contains an objectStore datastore is the following:

1 - You don't want to try to make a full webgisdr backup that includes an objectStore because as soon as you have some data (> 100 Go),
webgisdr just never finishes (cf. https://community.esri.com/t5/high-availability-and-disaster-recovery-questions/can-t-backup-arcgis-... - BUG-000173313 [1]). At least, this is my experience.
2 - Make a manual backup of your production objectStore using the "backupdatastore" command: `backupdatastore.bat --store object` (if backup drive destination is local, it only takes few hours...
BUG-000173313 is mainly due to the copie of hundred of millions files to the webgisdr SHARED_LOCATION and BACKUP_LOCATION which have to be shared drives accessible by all ArcGIS Enteprise components)
3- Make a webgisdr that not include the objectStore by setting "INCLUDE_GRAPH_STORE_DATA" to false.
4- On your standby environment configure all the VMs including the configuration of the objectStore
5- Run import of webgidr on the standby environment
6- Check webgisdr import was successful. On ArcGIS Manager, validate your datastores. If it loads forever, it very likely means that the query on the objectStore failed. You need to try again (BUG-000173247 [2]).
7 - Go on the ArcGIS Server admin interface, section "data/items/cloudStore" and check the machine(s) name:
https://myags.company.com:6443/arcgis/admin/data/items/cloudStores/AGSDataStore_objectstore_oz_aev7q
It unfortunately is the machines from the production environment. Your production object datatore is now "consummed" from the 2 ArcGIS Server environments.
8 - We now have to restore the objectStore using ArcGIS Datastore CLI if we want a truely replicated environment. First step is to disconnect the ojectStore from the replicated ArcGIS Server.
9 - Go to ArcGIS Server Manager in the data stores section and try to enregistrer it. You can't because of BUG-000172916 [3]. You need to do it from the ADMIN interface
10 - Go to ArcGIS Server Admin interface.
* Go https://myags.company.com:6443/arcgis/admin/data/items/cloudStores/ and copy the string of "Child items": something like "/cloudStores/AGSDataStore_objectstore_oz_aev7q1p"
* Go to https://myags.company.com:6443/arcgis/admin/data/unregisterItem and paste the copied string and click "Unregister"
* Go back to the "Manager" interface and check the objectStore is no longer there
11 - Detach the backup objectStore drive and attach it to the standby datastore VM so that you restore it using the CLI (or you can try from a shared drive but it will take forever...):
`restoredatastore.bat --store object --target most-recent --source-loc E:\arcgis\backup\object --data-dir D:\arcgis\arcgisdatastore --server-url https://myags.company.com:6443 --server-admin siteadmin --server-password updateme`
12 - After hours of restore, you should now have a functional truely replicated objectStore attached to the standby ArcGIS Server

[1] BUG-000173313 - After migrating hosted scene layer caches from the tile cache data store to the object store, backups of the object store take significantly longer than backups of the tile cache data store containing the same content in ArcGIS Enterprise.
[2] BUG-000173247 - Can't validate 2 times in a row the objectStore
[3] BUG-000172916 - The ArcGIS Data Store object data store cannot be unregistered in Server Manager using the unregister button

Needless to say, it is a difficult workflow to automate... I used to have daily sync overnight (5 hours) at 11.3, but now it is clearly impossible.

Anybody else suffering as well with the ojectStore backup ?

Thanks

[PhilippeVDV]

Thanks for sharing this valuable info @NicolasGIS ! I could not test yet with replication of the objectstore at 11.5 because we face an issue when upgrading ArcGIS Server from 11.2 to 11.5. But seems that there are a lot of issues with 11.5. See also what is mentioned here: https://www.reddit.com/r/gis/comments/1nk3ebw/arcgis_enterprise_after_upgrade_to_115_feature/ : "They [Esri] said to wait a bit longer, so they can iron out all the issues and release patches last I heard. 11.5 is an LTR, so it's a good one to go to when they get it sorted."