topic Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine in High Availability and Disaster Recovery Questions

11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

NicolasGIS — Sat, 27 Sep 2025 14:52:10 GMT

Hello,

I am testing the replication of an Enterprise 11.5 deployment on Windows Server 2022 with 3 datastores (tileCache, relational, and objectStore).

I set up a standby environment on a different set of VMs and restore a webgisdr backup. At first sight, everything is in order and all the components are properly restored and connected one another.

But when having a closer a look at the objectStore, I realized that is now pointing to the original objectStore.

You can notice it:

- from the ArcGIS Server manager when validating the objectStore: the _proxyUrl parameter is:

https://myoriginaldatastoreVM.company.com:2443/arcgis/datastoreadmin/machines/MYORIGINGALDATASTOREVM.COMPANY.COM/validate

- from the ArcGIS Server admin interface when checking the machines of the cloud store "objectStore":

https://myags.company.com:6443/arcgis/admin/data/items/cloudStores/AGSDataStore_objectstore_oz_aev7q1p/machines

it is the name of the original datastore VM which is written (ie: "myoriginaldatastore").

Also, if you now check the replicatedDataStore VM, and browse:

https://localhost:2443/arcgis/datastore/

You will see that it is no longer "attached" on the replicated deployment.

I am confused: is it a BUG or a feature ? Did I miss anything @JonathanQuinn maybe ?

Thanks,

/cc FYI @PhilippeVDV @CedricDespierreCorporon

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

NicolasGIS — Sun, 28 Sep 2025 16:14:01 GMT

And the worst is that it also happens if your webgidr does not include object store (by setting "INCLUDE_GRAPH_STORE_DATA" to false) !
So the only working workflow to fully replicate a 11.5 ArcGIS Enterprise deployment that contains an objectStore datastore is the following:

1 - You don't want to try to make a full webgisdr backup that includes an objectStore because as soon as you have some data (> 100 Go),
webgisdr just never finishes (cf. https://community.esri.com/t5/high-availability-and-disaster-recovery-questions/can-t-backup-arcgis-enterprise-11-4-webgisdr-since/td-p/1562990 - BUG-000173313 [1]). At least, this is my experience.
2 - Make a manual backup of your production objectStore using the "backupdatastore" command: `backupdatastore.bat --store object` (if backup drive destination is local, it only takes few hours...
BUG-000173313 is mainly due to the copie of hundred of millions files to the webgisdr SHARED_LOCATION and BACKUP_LOCATION which have to be shared drives accessible by all ArcGIS Enteprise components)
3- Make a webgisdr that not include the objectStore by setting "INCLUDE_GRAPH_STORE_DATA" to false.
4- On your standby environment configure all the VMs including the configuration of the objectStore
5- Run import of webgidr on the standby environment
6- Check webgisdr import was successful. On ArcGIS Manager, validate your datastores. If it loads forever, it very likely means that the query on the objectStore failed. You need to try again (BUG-000173247 [2]).
7 - Go on the ArcGIS Server admin interface, section "data/items/cloudStore" and check the machine(s) name:
https://myags.company.com:6443/arcgis/admin/data/items/cloudStores/AGSDataStore_objectstore_oz_aev7q
It unfortunately is the machines from the production environment. Your production object datatore is now "consummed" from the 2 ArcGIS Server environments.
8 - We now have to restore the objectStore using ArcGIS Datastore CLI if we want a truely replicated environment. First step is to disconnect the ojectStore from the replicated ArcGIS Server.
9 - Go to ArcGIS Server Manager in the data stores section and try to enregistrer it. You can't because of BUG-000172916 [3]. You need to do it from the ADMIN interface
10 - Go to ArcGIS Server Admin interface.
* Go https://myags.company.com:6443/arcgis/admin/data/items/cloudStores/ and copy the string of "Child items": something like "/cloudStores/AGSDataStore_objectstore_oz_aev7q1p"
* Go to https://myags.company.com:6443/arcgis/admin/data/unregisterItem and paste the copied string and click "Unregister"
* Go back to the "Manager" interface and check the objectStore is no longer there
11 - Detach the backup objectStore drive and attach it to the standby datastore VM so that you restore it using the CLI (or you can try from a shared drive but it will take forever...):
`restoredatastore.bat --store object --target most-recent --source-loc E:\arcgis\backup\object --data-dir D:\arcgis\arcgisdatastore --server-url https://myags.company.com:6443 --server-admin siteadmin --server-password updateme`
12 - After hours of restore, you should now have a functional truely replicated objectStore attached to the standby ArcGIS Server

[1] BUG-000173313 - After migrating hosted scene layer caches from the tile cache data store to the object store, backups of the object store take significantly longer than backups of the tile cache data store containing the same content in ArcGIS Enterprise.
[2] BUG-000173247 - Can't validate 2 times in a row the objectStore
[3] BUG-000172916 - The ArcGIS Data Store object data store cannot be unregistered in Server Manager using the unregister button

Needless to say, it is a difficult workflow to automate... I used to have daily sync overnight (5 hours) at 11.3, but now it is clearly impossible.

Anybody else suffering as well with the ojectStore backup ?

Thanks

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

PhilippeVDV — Mon, 29 Sep 2025 14:33:38 GMT

Thanks for sharing this valuable info @NicolasGIS ! I could not test yet with replication of the objectstore at 11.5 because we face an issue when upgrading ArcGIS Server from 11.2 to 11.5. But seems that there are a lot of issues with 11.5. See also what is mentioned here: https://www.reddit.com/r/gis/comments/1nk3ebw/arcgis_enterprise_after_upgrade_to_115_feature/ : "They [Esri] said to wait a bit longer, so they can iron out all the issues and release patches last I heard. 11.5 is an LTR, so it's a good one to go to when they get it sorted."

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

NicolasGIS — Tue, 14 Oct 2025 05:42:29 GMT

BUG-000180128 - After disaster recovery with webgisdr restore in new passive stand by environment, ArcGIS Data Store object store host name is pointing still to the old object store host name in both ArcGIS Data Store (describedatastore) and ArcGIS Server Data Stores.

Status: In Product plan... Wait and see

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

NicolasGIS — Fri, 28 Nov 2025 15:28:04 GMT

Still not there in 12.0 😕

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

SimonSchütte_ct — Tue, 02 Dec 2025 13:27:18 GMT

Same issue. Mission critical.
HA multi-stage sync with object store cluster - Esri Community HA multi-stage sync with object store cluster - Esri Community

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

PhilippeVDV — Fri, 13 Feb 2026 10:39:15 GMT

@NicolasGIS , @SimonSchütte_ct

First time that I'm using webgisdr in combination with Object store.

When doing the export on the active environment I don't see a reference to the object store. See attached screenshot

Is this normal?

Object store appears in ArcGIS Server Manager

Best regards

Phil

UPDATE: OK, found it. I was still using an old version (11.2) of the webgisdr.properties file.

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

NicolasGIS — Fri, 13 Feb 2026 10:32:34 GMT

Did you configure include OBJECT STORE in webgisdr config file ?

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

SimonSchütte_ct — Thu, 23 Apr 2026 12:41:33 GMT

BUG-000180128 for ArcGIS Data Store
Version fixed: 12.1
Patch for 11.5: ?

Workarounds:

"Migrate to use cloud storage or modify the URL of the ArcGIS Data Store object store in the Admin API."
Addition from Esri Support:
"Modify the URL of the ArcGIS Data Store object store in the Admin API. The location is in ArcGIS Server Admin page. Home>data>items>cloudStores>[ArcGIS Datastore Object Store]>edit.
The following information needs to be edited: “name” , “endpoint” and “adminURL”."
Note that this is a manual step, which needs to be repeated with each import.

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

PhilippeVDV — Fri, 13 Feb 2026 11:22:31 GMT

Apparently I was still using an 11.2 webgisdr.properties file where this parameter was not present yet. So this explains why object store was not included in the export. However, I understand from your explanation that it is better to take a separate backup from object store.

Also 'funny' to read all the BUGS here that you encountered so far with object store. I also ran into those (validation of object store that hangs, no able to unregister it from arcgis server manager). As we also replicate to a standby environment I will need to follow that workaround as well. Thanks for sharing this valuable information @NicolasGIS !

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

PhilippeVDV — Mon, 02 Mar 2026 16:13:46 GMT

Apparently it will be fixed in 12.1: BUG-000180128 for ArcGIS Data Store

Let's hope so

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

PhilippeVDV — Wed, 22 Apr 2026 08:03:42 GMT

@NicolasGIS @SimonSchütte_ct

This one seems to be solved through the new released patch: https://support.esri.com/en-us/patches-updates/2026/arcgis-server-security-2026-update-1-patch

Can you confirm that the patch solves it indeed?

(I'm still on 11.5, but curious whether the patch solves it in 12.0)

Best regards

Philippe

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

SimonSchütte_ct — Thu, 23 Apr 2026 12:46:53 GMT

@PhilippeVDV I had the opportunity to test this part of the patch prior public release and it worked as intended. I´d highly recommend installing all security patches anyway 🙂

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

SimonSchütte_ct — Fri, 24 Apr 2026 07:22:27 GMT

"This issue is resolved with the ArcGIS Server Security 2026 Update 1 Patch. The URL is: https://support.esri.com/en-us/patches-updates/2026/arcgis-server-security-2026-update-1-patch Patches are available for versions 12.0, 11.5, 11.4, 11.3 and 11.1. Refer to the 'Issues addressed with this patch' section of the patch page to determine the version(s) where this defect is addressed."
BUG-000180128 for ArcGIS Data Store

BUG-000180128 - After disaster recovery with the WebGISDR restore in a new passive standby environment, the ArcGIS Data Store object store host name is still pointing to the old object store host name in both ArcGIS Data Store (describedatastore) and ArcGIS Server data stores.

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

PhilippeVDV — Thu, 23 Apr 2026 11:27:59 GMT

OK, thanks for the confirmation @SimonSchütte_ct . The patch notes are bit confusing according to me: ArcGIS Server Security 2026 Update 1 Patch

"BUG-000180128 - After disaster recovery with the WebGISDR restore in a new passive standby environment, the ArcGIS Data Store object store host name is still pointing to the old object store host name in both ArcGIS Data Store (describedatastore) and ArcGIS Server data stores. (12.0 only)"

This made me think that only the patch for 12.0 solves it in AGE 12.0. But I understand from your observations that the 11.5 patch solves it in 11.5 as well?

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

SimonSchütte_ct — Thu, 23 Apr 2026 12:41:05 GMT

Yes, this is confusing and contradicts what is written below the bug details. I am not sure what to make of this.

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

NicolasGIS — Fri, 24 Apr 2026 05:50:21 GMT

Hi @PhilippeVDV and @SimonSchütte_ct,

I deployed the patch on 11.5 environment and I confirm it does fix the bug 🙂 !

Re: 11.5 - webgisdr - replication of objectStore broken - still pointing to original machine

PhilippeVDV — Mon, 27 Apr 2026 13:05:49 GMT

I installed the patch today and indeed, this one is solved