WEBGISDR not functioning @12.0 with cloud object store?

Question

Hi there,

I have a deployment using Azure blob for the cloud object store @ 12.0. I am seeing ArcGIS Server backups fail once they get over a certain size, or if I attempt to include object store caches in the backups. I get this message in the webgisdr logs:

{
"name": "ArcGIS Server",
"messages": "Cannot access the backup file for ArcGIS Server in webgisdr-backup/12-0-0/backup/20260505-091107-backup\server\cd67fdf9jcwpzzk6.",
"URL": "https://xyzxyzxy/hosting",
"status": "failure"
}

Latest job status of async backup operation on server https://xyzxyzx/hosting: {"jobid":"j65e79912-4aaa-46d3-8b8d-0717760d431b","stepInfo":["Export operation started.","Export operation failed."],"status":"EXECUTING"}.
"{"jobid":"j65e79912-4aaa-46d3-8b8d-0717760d431b","status":"COMPLETED","stepInfo":["Export operation started.","Export operation failed."],"operationResponse":{"status":"error","messages":["Export operation failed. Failed to write the file '20260505-091124.agssite' to this location 'webgisdr-backup/12-0-0/backup/20260505-091107-backup/server/cd67fdf9jcwpzzk6'"]}}"

looking at a manual export of that site from the admin API I get:

Failed to write the file '20260505-095011.agssite' to this location 'backup-20260504' Failed to upload backup-20260504/20260505-095011.agssite to object store.. Caused by: Timeout on blocking read for 300000000000 NANOSECONDS
Export operation failed. Failed to write the file '20260505-095011.agssite' to this location 'backup-20260504'

which is an exception in:

Caused by: Timeout on blocking read for 300000000000 NANOSECONDS at com.esri.arcgis.objectstore.clients.azure.AzureBlobStoreClient.putObject(AzureBlobStoreClient.java:986) at com.esri.arcgis.objectstore.clients.azure.AzureBlobStoreClient.putObject(AzureBlobStoreClient.java:915) at com.esri.arcgis.discovery.admin.backuprestore.BackupRestoreUtil.writeSiteFileToBackupStore(BackupRestoreUtil.java:888)

This worked at 11.5, there was a change from staging everything locally first before uploading to writing directly to blob storage at 12.0. We are using private endpoints to access the storage and should have a clear path to it.

I have seen two bugs with 12.0 but don't know if this is related to what I'm seeing:

https://support.esri.com/en-us/bug/large-web-gis-disaster-recovery-webgisdr-exports-fail-t-bug-000176368

https://support.esri.com/en-us/bug/after-migrating-hosted-scene-layer-caches-from-the-tile-bug-000173313

If anyone has hit this have you got a cause / solution? I'm going to staging locally and uploading to blob in the meantime.

Thanks!

malte · Answer

Hi, I think that was the problem in my case. I'm on 11.5, the object store is on-prem with the dedicated object store role for datastore. I had an azure cloud store attached to the arcgis server, but for caches, not as an object store. I think it assumed it's an object store as well and tries to back it up. When I removed the azure store (it was for testing purposes only), it worked.

FraserHand · Answer

Also note there is an issue backing up to blob storage with big webgisdr files where this times out. This I think is fixed in 12.1 and I think there is a patch coming for 12.0.

FraserHand · Answer

Hi, unfortunately no answer yet though I have logged an issue with Esri. What version are you at? At 12.0 backing up the object store seemed busted - I tested under 12.1 and it was backing up again ok. There is also an issue if you register a cloud store (say for cache locations) at 12.0 where it thinks it should be an object store - I haven't tested that at 12.1.