Can't backup ArcGIS Enterprise 11.4 webgisdr since migrated to from tileCache to objectStore

[NicolasGIS]

Hello,

Testing the upgrade from ArcGIS Enterprise 11.3 running on Windows Server 2022 to 11.4 and I am facing an issue after having migrated my tileCache datastore to objectStore using the `MigrateSceneServices` utility.

https://enterprise.arcgis.com/en/server/latest/publish-services/windows/migrate-scene-services-utili...

Here is the workflow:

- From 11.3, upgrade to 11.4 OK

- Export webgisdr backup OK

- Import webgisdr backup OK

- Upgrade scene services using `MigrateSceneServices` utility: OK

- Update the webgisdr.properties to INCLUDE_OBJECT_STORE_CACHES = true and INCLUDE_SCENE_TILES_CACHES = false

- Run the webgisdr export

It systematically fails after 24 hours... (before it was done in 1 hour !):

The tileCache was about 29 Go and composed of 20 scene services (mainly meshes).

Anybody else faced the same issue ?

Thanks,

Nicolas

/cc @JonathanQuinn 

[JonathanQuinn]

Once migrated, can you check the size of the object store folder (for example C:\arcgisdatastore\ozonedata); is it also 29GB? When you are waiting for the backup to be created, can you check whether the size of the temporary folder for the object store backup is growing in size, and how fast? It'll be a path under your SHARED_LOCATION path, with a guid. You'll need to look in each folder with the guid to see if its the relational or object store backup.

[NicolasGIS]

I have the intuition that there are so many files to copy that it takes ages but that is just a feeling...

Only retrieving the size of the folder the 'ozonedata' took couple of hours as there are so many files. Small comparison:

- before migration 'nosqldata' folder:  27.7 GB, 265 files, 9 folders

- after migration 'ozonedata' folder:   21.4 GB, 2 353 27 files, 81 folders

When I am waiting for the backup to be made, I can tell that the 'local' object store backup works (ie: the first backup which is done in the configured backup location). Using the ArcGIS Datastore utility command 'listbackups', I can tell it is running during around 9 hours (!), then I can see the backup listed as SUCCESS using this utility. Finally, it starts backuping over to the SHARED_LOCATION but the copy of the object store is very slow: it is growing super slowy.

In the end, I think it only fails because it reached the hard coded time out value of 24 hours. It is copying slowly but surely but is very inefficient due to the large number of files. When it fails after 24 hours, the objectStore backup folder is around 9 Go only so there is still a lot to copy remaining.

I suspected a network drive performance issue so I moved the SHARED_LOCATION folder and BACKUP folders of the webgisdr to a local drive on the machine (it is a base deployment for testing purpose) and it failed as well. I am now requesting a new drive with more IO to see if it helps..

It kind of reminds me this issue related to the TILE_CACHE and switch to higher IO drive had helped: https://community.esri.com/t5/high-availability-and-disaster-recovery-questions/tilecache-datastore-...

Thanks

[Gaius_Kuttappan]

Hello @NicolasGIS

I’ve gone through the article "Migrate hosted scene layer caches using the MigrateSceneServices utility" which mentions that the disk space used by the object store folder should be the same as the tile cache store. Based on the figures you shared, I just wanted to double-check the following basics:

• Is there any reduction in size after the migration? Are all the migrated scene layers loading correctly?
• Additionally, we could verify if the data stores are validating properly.
• Run the describedatastore command to check for any issues. ReadWrite mode?
• Does the domain or local account running services have full permissions on all content folders and webgisdr locations?
• Have you seen anything in the webgisdr logs?

Thanks!

[NicolasGIS]

Hello,

Apologize for my late reply but tests are taking ages....
I will summarize my testings but here is my conclusion: everything seems to be working properly but it is just super long and slow.

I found out that "Windows Defender" was slowing the backup down. I added exclusions to:
- arcgis datastore backup folder

- webgisdr temp and backup folder

and after that, I was able to make a webgisdr backup... but performances are really poor compared to tileCache. Here is a quick overview:

Scenario 1:

-------------------

- 1 base deployment - Portal for ArcGIS, ArcGIS for Server, ArcGIS datastores (relational, tileCache, objectStore) - Windows 2022 16 CPU - 60 GB RAM deployment - Disk: 160 GB.

- Attached local D drive "io3" 300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write)

Scenario 2 (objectStore on a dedicated VM):

-------------------

- 1 base deployment - Portal for ArcGIS, ArcGIS for Server, ArcGIS datastores (relational, tileCache) - Windows 2022 16 CPU - 60 GB RAM deployment - Disk: 160 GB.

- 1 datastore deployment - ArcGIS datastores  (objectStore) - Windows 2022 16 CPU - 60 GB RAM deployment - Disk: 160 GB.

- Attached local D drive "io3" 300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write)

As you can see it now takes:

- 12h30 in a base deployment when webgisdr temp and backup folder can be local

- 16h in a scenario with a shared webgisdr temp and backup folder

(we can ignore the deletion but thought it is interesting to note that it is now taking ages as well)

I am bit scared now because this ArcGIS Enterprise deployment is small. TileCache is only 27 Go. Our professional WebGIS has currently a tileCache of 110 Go and I am not sure it will fit in 24 hours. I am about to start the test.

It seems to me that with the objectStore, we are retrograding back in time of BUG-000139154

BUG-000139154: https://support.esri.com/en-us/bug/tile-cache-datastore-backup-takes-too-long-and-the-data-bug-00013...

The bad piece of new is that there is only room for 1 version to fix it (11.5) !

Note that a webgisdr backup of the same site on tileCache takes only 30 minutes !

And finally to answer your question @Gaius_Kuttappan :

- There is a size reduction after migration (from 27.8 Go -> 21. 4GB)  and all migrated scene layers are loading correctly

- Yes, datastore are validating properly

- Describedatastore: READWRITE - nothing special

- Yes, full permissions correctly set.

- Nothing abnormal in webgisdr logs as everything is working as expected.

Any thought @JonathanQuinn ?

Thanks !

[JonathanQuinn]

I was able to reproduce significant slow-down in the backup of the object store:

1. Created 20GB of data in the tile cache data store at 11.3
2. Created a backup, which took 22 minutes or so
3. Upgraded to 11.4
4. Migrating the data to the object store
5. Created a backup and it took 3.5 hours

I did notice that Windows anti-virus was certainly busy; there are many more individual files that are copied, but I created an internal issue to investigate. If you want something to track, reach out to Support to log a bug.

[NicolasGIS · Accepted solution]

Thanks @JonathanQuinn  for your fast reply and test: much appreciated.

Local support was also able to reproduce the issue and logged the following bug:

BUG-000173313 - After migrating hosted scene layer caches from the tile cache data store to the object store, backups of the object store take significantly longer than backups of the tile cache data store containing the same content in ArcGIS Enterprise.

The good piece of news is that it is already "In Product Plan".

What a relief because as I was afraid, I cannot backup my 110 Go objectStore (it took 21 hours to do the local backup and then webgisdr timed-out 3 hours later having started to copy the backup to the shared location)