WEBGISDR not functioning @12.0 with cloud object store?

[FraserHand]

Hi there,

I have a deployment using Azure blob for the cloud object store @ 12.0. I am seeing ArcGIS Server backups fail once they get over a certain size, or if I attempt to include object store caches in the backups. I get this message in the webgisdr logs:

{
"name": "ArcGIS Server",
"messages": "Cannot access the backup file for ArcGIS Server in webgisdr-backup/12-0-0/backup/20260505-091107-backup\\server\\cd67fdf9jcwpzzk6.",
"URL": "https://xyzxyzxy/hosting",
"status": "failure"
}

Latest job status of async backup operation on server https://xyzxyzx/hosting: {"jobid":"j65e79912-4aaa-46d3-8b8d-0717760d431b","stepInfo":["Export operation started.","Export operation failed."],"status":"EXECUTING"}.
"{"jobid":"j65e79912-4aaa-46d3-8b8d-0717760d431b","status":"COMPLETED","stepInfo":["Export operation started.","Export operation failed."],"operationResponse":{"status":"error","messages":["Export operation failed. Failed to write the file '20260505-091124.agssite' to this location 'webgisdr-backup/12-0-0/backup/20260505-091107-backup/server/cd67fdf9jcwpzzk6'"]}}"

looking at a manual export of that site from the admin API I get:

Failed to write the file '20260505-095011.agssite' to this location 'backup-20260504' Failed to upload backup-20260504/20260505-095011.agssite to object store.. Caused by: Timeout on blocking read for 300000000000 NANOSECONDS
Export operation failed. Failed to write the file '20260505-095011.agssite' to this location 'backup-20260504'

which is an exception in:

Caused by: Timeout on blocking read for 300000000000 NANOSECONDS at com.esri.arcgis.objectstore.clients.azure.AzureBlobStoreClient.putObject(AzureBlobStoreClient.java:986) at com.esri.arcgis.objectstore.clients.azure.AzureBlobStoreClient.putObject(AzureBlobStoreClient.java:915) at com.esri.arcgis.discovery.admin.backuprestore.BackupRestoreUtil.writeSiteFileToBackupStore(BackupRestoreUtil.java:888)

This worked at 11.5, there was a change from staging everything locally first before uploading to writing directly to blob storage at 12.0. We are using private endpoints to access the storage and should have a clear path to it.

I have seen two bugs with 12.0 but don't know if this is related to what I'm seeing:

https://support.esri.com/en-us/bug/large-web-gis-disaster-recovery-webgisdr-exports-fail-t-bug-00017...

https://support.esri.com/en-us/bug/after-migrating-hosted-scene-layer-caches-from-the-tile-bug-00017...

If anyone has hit this have you got a cause / solution? I'm going to staging locally and uploading to blob in the meantime.

Thanks!

[FraserHand]

UPDATE: I'm trying to write the backup to a local drive / share now and have updated the webgisdr properties to suit. If I run with no Azure config in the webgisdr.properties file I get this:

{"backupLocation":"","messages":"The value of the property BACKUP_BLOB_ACCOUNT_NAME cannot be empty.","id":"cbcfbc64699144a89586a1bb1383ba80","operation":"backup","results":[],"status":"failure","elapsedTime":"00hr:00min:02sec"}

and if I put in the location of the objstore container I get this in the server logs and the export fails:

Could not connect to the Backup Object Store specified. Check if the specified parameters are valid. JSONObject["objectStore"] not found.

this is with INCLUDE_OBJECT_STORE_CACHES = true

[FraserHand]

Another update - if I set:

# Specify whether to include object store caches.
INCLUDE_OBJECT_STORE_CACHES = false

then I can get a .agssite file locally.

[malte]

Hi,

did you find a solution to backup with the object store?

I am currently having the same issue. It seems like the tool assumes the object store is a cloud store. In the server admin, the cloud store is also listed under cloudStores: server/admin/data/items/cloudStores/AGSDataStore_objectstore_oz_suwxku9

[FraserHand]

Hi, unfortunately no answer yet though I have logged an issue with Esri. What version are you at? At 12.0 backing up the object store seemed busted - I tested under 12.1 and it was backing up again ok. There is also an issue if you register a cloud store (say for cache locations) at 12.0 where it thinks it should be an object store - I haven't tested that at 12.1. 

[malte]

Hi, I think that was the problem in my case. I'm on 11.5, the object store is on-prem with the dedicated object store role for datastore. I had an azure cloud store attached to the arcgis server, but for caches, not as an object store. I think it assumed it's an object store as well and tries to back it up. When I removed the azure store (it was for testing purposes only), it worked.

[FraserHand]

Also note there is an issue backing up to blob storage with big webgisdr files where this times out. This I think is fixed in 12.1 and I think there is a patch coming for 12.0.