topic Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore in High Availability and Disaster Recovery Questions

Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

NicolasGIS — Wed, 27 Nov 2024 13:24:41 GMT

Hello,

Testing the upgrade from ArcGIS Enterprise 11.3 running on Windows Server 2022 to 11.4 and I am facing an issue after having migrated my tileCache datastore to objectStore using the `MigrateSceneServices` utility.

https://enterprise.arcgis.com/en/server/latest/publish-services/windows/migrate-scene-services-utility.htm

Here is the workflow:

- From 11.3, upgrade to 11.4 OK

- Export webgisdr backup OK

- Import webgisdr backup OK

- Upgrade scene services using `MigrateSceneServices` utility: OK

- Update the webgisdr.properties to INCLUDE_OBJECT_STORE_CACHES = true and INCLUDE_SCENE_TILES_CACHES = false

- Run the webgisdr export

It systematically fails after 24 hours... (before it was done in 1 hour !):

The tileCache was about 29 Go and composed of 20 scene services (mainly meshes).

Anybody else faced the same issue ?

Thanks,

Nicolas

/cc @JonathanQuinn

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

JonathanQuinn — Wed, 27 Nov 2024 15:44:55 GMT

Once migrated, can you check the size of the object store folder (for example C:\arcgisdatastore\ozonedata); is it also 29GB? When you are waiting for the backup to be created, can you check whether the size of the temporary folder for the object store backup is growing in size, and how fast? It'll be a path under your SHARED_LOCATION path, with a guid. You'll need to look in each folder with the guid to see if its the relational or object store backup.

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

NicolasGIS — Thu, 28 Nov 2024 06:51:30 GMT

I have the intuition that there are so many files to copy that it takes ages but that is just a feeling...

Only retrieving the size of the folder the 'ozonedata' took couple of hours as there are so many files. Small comparison:

- before migration 'nosqldata' folder: 27.7 GB, 265 files, 9 folders

- after migration 'ozonedata' folder: 21.4 GB, 2 353 27 files, 81 folders

When I am waiting for the backup to be made, I can tell that the 'local' object store backup works (ie: the first backup which is done in the configured backup location). Using the ArcGIS Datastore utility command 'listbackups', I can tell it is running during around 9 hours (!), then I can see the backup listed as SUCCESS using this utility. Finally, it starts backuping over to the SHARED_LOCATION but the copy of the object store is very slow: it is growing super slowy.

In the end, I think it only fails because it reached the hard coded time out value of 24 hours. It is copying slowly but surely but is very inefficient due to the large number of files. When it fails after 24 hours, the objectStore backup folder is around 9 Go only so there is still a lot to copy remaining.

I suspected a network drive performance issue so I moved the SHARED_LOCATION folder and BACKUP folders of the webgisdr to a local drive on the machine (it is a base deployment for testing purpose) and it failed as well. I am now requesting a new drive with more IO to see if it helps..

It kind of reminds me this issue related to the TILE_CACHE and switch to higher IO drive had helped: https://community.esri.com/t5/high-availability-and-disaster-recovery-questions/tilecache-datastore-very-slow-to-restore/td-p/1121135/page/2

Thanks

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

Gaius_Kuttappan — Wed, 04 Dec 2024 09:45:22 GMT

Hello @NicolasGIS

I’ve gone through the article "Migrate hosted scene layer caches using the MigrateSceneServices utility" which mentions that the disk space used by the object store folder should be the same as the tile cache store. Based on the figures you shared, I just wanted to double-check the following basics:

Is there any reduction in size after the migration? Are all the migrated scene layers loading correctly?
Additionally, we could verify if the data stores are validating properly.
Run the describedatastore command to check for any issues. ReadWrite mode?
Does the domain or local account running services have full permissions on all content folders and webgisdr locations?
Have you seen anything in the webgisdr logs?

Thanks!

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

NicolasGIS — Thu, 12 Dec 2024 10:39:48 GMT

Hello,

Apologize for my late reply but tests are taking ages....
I will summarize my testings but here is my conclusion: everything seems to be working properly but it is just super long and slow.

I found out that "Windows Defender" was slowing the backup down. I added exclusions to:
- arcgis datastore backup folder

- webgisdr temp and backup folder

and after that, I was able to make a webgisdr backup... but performances are really poor compared to tileCache. Here is a quick overview:

Scenario 1:

-------------------

- 1 base deployment - Portal for ArcGIS, ArcGIS for Server, ArcGIS datastores (relational, tileCache, objectStore) - Windows 2022 16 CPU - 60 GB RAM deployment - Disk: 160 GB.

- Attached local D drive "io3" 300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write)

Scenario 2 (objectStore on a dedicated VM):

-------------------

- 1 base deployment - Portal for ArcGIS, ArcGIS for Server, ArcGIS datastores (relational, tileCache) - Windows 2022 16 CPU - 60 GB RAM deployment - Disk: 160 GB.

- 1 datastore deployment - ArcGIS datastores (objectStore) - Windows 2022 16 CPU - 60 GB RAM deployment - Disk: 160 GB.

- Attached local D drive "io3" 300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write)

As you can see it now takes:

- 12h30 in a base deployment when webgisdr temp and backup folder can be local

- 16h in a scenario with a shared webgisdr temp and backup folder

(we can ignore the deletion but thought it is interesting to note that it is now taking ages as well)

I am bit scared now because this ArcGIS Enterprise deployment is small. TileCache is only 27 Go. Our professional WebGIS has currently a tileCache of 110 Go and I am not sure it will fit in 24 hours. I am about to start the test.

It seems to me that with the objectStore, we are retrograding back in time of BUG-000139154

BUG-000139154: https://support.esri.com/en-us/bug/tile-cache-datastore-backup-takes-too-long-and-the-data-bug-000139154

The bad piece of new is that there is only room for 1 version to fix it (11.5) !

Note that a webgisdr backup of the same site on tileCache takes only 30 minutes !

And finally to answer your question @Gaius_Kuttappan :

- There is a size reduction after migration (from 27.8 Go -> 21. 4GB) and all migrated scene layers are loading correctly

- Yes, datastore are validating properly

- Describedatastore: READWRITE - nothing special

- Yes, full permissions correctly set.

- Nothing abnormal in webgisdr logs as everything is working as expected.

Any thought @JonathanQuinn ?

Thanks !

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

JonathanQuinn — Thu, 12 Dec 2024 16:52:12 GMT

I was able to reproduce significant slow-down in the backup of the object store:

Created 20GB of data in the tile cache data store at 11.3
Created a backup, which took 22 minutes or so
Upgraded to 11.4
Migrating the data to the object store
Created a backup and it took 3.5 hours

I did notice that Windows anti-virus was certainly busy; there are many more individual files that are copied, but I created an internal issue to investigate. If you want something to track, reach out to Support to log a bug.

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

NicolasGIS — Thu, 02 Jan 2025 18:19:22 GMT

Thanks @JonathanQuinn for your fast reply and test: much appreciated.

Local support was also able to reproduce the issue and logged the following bug:

BUG-000173313 - After migrating hosted scene layer caches from the tile cache data store to the object store, backups of the object store take significantly longer than backups of the tile cache data store containing the same content in ArcGIS Enterprise.

The good piece of news is that it is already "In Product Plan".

What a relief because as I was afraid, I cannot backup my 110 Go objectStore (it took 21 hours to do the local backup and then webgisdr timed-out 3 hours later having started to copy the backup to the shared location)

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

NicolasGIS — Tue, 16 Sep 2025 08:45:53 GMT

Hello @JonathanQuinn ,

So 11.5 is out for 4 months now. BUG remains: no patch. BUG was already reported in 11.4 and 11.5 beta testing phase.

11.5 is the last version supporting tileCache. New cache has to be published in objectStore. What is your roadmap for BUG-000173313 ? Are you aware this is really problematic ? Are we supposed to escalade it once again ? To me functional backup/restore non functional should be by default your top priority...

Any updates ?

Thanks.

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

PhilippeVDV — Mon, 29 Sep 2025 07:56:36 GMT

I fully agree with you @NicolasGIS . Backup/restore and replication through webgisdr is an essential workflow and when it appears to be buggy, it should get the highest priority to get fixed!

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

NicolasGIS — Fri, 28 Nov 2025 15:28:46 GMT

Seems like it is still not fixed at 12.0 😕

https://content.esri.com/support/downloads/other_/120-issuesaddressedlist_11192025.pdf

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

Marc_Graham — Wed, 01 Apr 2026 22:52:58 GMT

Hi there,

We just upgraded from 11.3 to 11.5 and migrated our tileCache to object Store.

Our WebGIS DR backups have gone from just over 8 hours to approximately 87 hours. Same data volume, same 3D content.

11.3 with Tile Cache:

2026-03-24 21:00:16 INFO [main] com.esri.arcgis.webgis.client.WebGISDR - Starting the WebGIS DR utility.

2026-03-25 05:13:39 INFO [main] com.esri.arcgis.webgis.client.WebGISDR - The WebGIS DR utility completed successfully.

11.5 With Object Store:

2026-03-29 19:32:30 INFO [main] com.esri.arcgis.webgis.client.WebGISDR - Starting the WebGIS DR utility.

2026-04-02 11:19:36 INFO [main] com.esri.arcgis.webgis.client.WebGISDR - The WebGIS DR utility completed successfully.

Clearly this is untenable. It is a business risk as we cannot finish a backup over the weekend and having WebGIS DR running for half of the week is not ideal.

@JonathanQuinn can we please get some information about when this might be improved. You just had Dev Summit, was it discussed there?Thanks,

Marc

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

Geoeki — Wed, 15 Apr 2026 07:54:15 GMT

"The good piece of news is that it is already "In Product Plan"."

This aged like milk.

Upgraded to 11.5 last weekend, same issue still. I'd also like to ask for an update please @JonathanQuinn

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

SimonSchütte_ct — Wed, 15 Apr 2026 12:02:09 GMT

@Marc_Graham I´d recommend opening a case in My Esri. You can escalate the case if needed.

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

NicolasGIS — Wed, 15 Apr 2026 12:09:02 GMT

Yes please! Ask to be attached to BUG-000173313 and then escalate to complain how critical it is ! It’s still not fixed in 12.1 so already 3 versions after it’s been posted here. Need more users complaining I guess!

Re: Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

Marc_Graham — Sun, 19 Apr 2026 23:14:36 GMT

We have done this also.

Very frustrating situation.