webgisdr Failed to export site - "A device attached to the system is not functioning"

[JamesGough]

We have WEBGISDR scheduled to run daily for our QA Portal environment. This is a high-availability multi-machine deployment.

The backup has been running without errors for months, but starting three days ago it began failing. Below is the full log:

2021-07-16 04:00:53 INFO  [main] com.esri.arcgis.webgis.client.WebGISDR - Starting the WebGIS DR utility.
2021-07-16 04:00:55 INFO  [main] com.esri.arcgis.webgis.storageservice.file.FileStorageService - Initializing File Storage Service with Base Directory: \\<network share path>\backup_location
2021-07-16 04:01:09 INFO  [pool-2-thread-3] com.esri.arcgis.webgis.component.service.impl.ServerDRService - The following ArcGIS Server has been backed up successfully: 
Url: https://gis.domain.com/raster.
2021-07-16 04:01:10 INFO  [pool-2-thread-7] com.esri.arcgis.webgis.component.service.impl.ServerDRService - The following ArcGIS Server has been backed up successfully: 
Url: https://gis.domain.com/geoevent.
2021-07-16 04:01:10 INFO  [pool-2-thread-4] com.esri.arcgis.webgis.component.service.impl.ServerDRService - The following ArcGIS Server has been backed up successfully: 
Url: https://gis.domain.com/image.
2021-07-16 04:01:11 INFO  [pool-2-thread-8] com.esri.arcgis.webgis.component.service.impl.ServerDRService - The following ArcGIS Server has been backed up successfully: 
Url: https://gis.domain.com/geoanalytics.
2021-07-16 04:01:21 INFO  [pool-2-thread-5] com.esri.arcgis.webgis.component.service.impl.ServerDRService - The following ArcGIS Server has been backed up successfully: 
Url: https://gis.domain.com/server.
2021-07-16 04:01:25 INFO  [pool-2-thread-1] com.esri.arcgis.webgis.component.service.impl.DataStoreDRService - The following Relational ArcGIS Data Store has been backed up successfully: 
Url: https://<datastore-internal-url>:2443/arcgis.
2021-07-16 04:01:27 INFO  [main] com.esri.arcgis.webgis.util.WebGISUtil - The backup of ArcGIS Data Store has completed in 00hr:00min:19sec.
2021-07-16 04:02:27 ERROR [pool-2-thread-6] com.esri.arcgis.webgis.component.service.impl.ServerDRService - Failed to back up the ArcGIS Server: 
Url: https://gis.domain.com/gis.
2021-07-16 04:02:27 ERROR [pool-2-thread-6] com.esri.arcgis.webgis.component.service.impl.ServerDRService - {"code":500,"messages":["Export operation failed. \\\\<network share path>\\sharedlocation\\WebGISSite1626422455003\\server\\7lf9iaqk5p2rn1nt (A device attached to the system is not functioning)"],"status":"error"}
2021-07-16 04:02:32 INFO  [main] com.esri.arcgis.webgis.util.WebGISUtil - The backup of ArcGIS Server has taken 00hr:01min:24sec.
2021-07-16 04:10:10 ERROR [pool-2-thread-2] com.esri.arcgis.webgis.component.service.impl.PortalDRService - Failed to back up the Portal for ArcGIS: 
Url: https://gis.domain.com/portal.
2021-07-16 04:10:10 ERROR [pool-2-thread-2] com.esri.arcgis.webgis.component.service.impl.PortalDRService - {"error":{"code":500,"details":null,"message":"Failed to export site. \\\\<network share path>\\sharedlocation\\WebGISSite1626422455003\\portal (A device attached to the system is not functioning)"}}
2021-07-16 04:10:10 INFO  [main] com.esri.arcgis.webgis.util.WebGISUtil - The backup of Portal for ArcGIS has taken 00hr:09min:02sec.
2021-07-16 04:10:10 INFO  [main] com.esri.arcgis.webgis.service.impl.WebGISDRFrontController - Deleting the temp directory \\<network share path>\sharedlocation\WebGISSite1626422455003.
2021-07-16 04:10:10 INFO  [main] com.esri.arcgis.webgis.util.WebGISUtil - The backup of Web GIS components has completed in 00hr:09min:15sec.
2021-07-16 04:10:10 INFO  [main] com.esri.arcgis.webgis.client.WebGISDR - Stopping the WebGIS DR utility.

What does the error "A device attached to the system is not functioning" mean?

[Anonymous User]

I would highly recommend completing the backup on the portal server locally and then moving to a file share.  This really reduces the backup time and sidesteps and communication or network issues.

[JamesGough]

Doesn't the shared location have to be a network share that all of the servers in the deployment can write to?

[Anonymous User · Accepted solution]

You can share a drive on the portal server with the others.  Just adjust the sharing properties and make sure your firewall rules allow.  My webgisdr creates the backup on the portal VM and then when the backup is complete a script moves it.  This method is much quicker and reduces the chance of network issues causing a failure.

[JamesGough]

I'll give that a try. Thanks.

[DavidHoy · Accepted solution]

The log says it took 1 min 24 sec to run the serversite export on the "gis" server site, and then failed when webgisdr tried to compress/copy the output file to the "shared_location".

In parallel, the portal backup process ran for 9 min 2 sec and then failed when webgisdr tried to compress/copy the output file to a different folder in the same "shared_location".

Possible reasons:

1. the fileserver hosting the "shared_location" ran out of disk space.
   • as webgisdr deletes all the temp output when it fails - it can be hard to see this when looking at the file server after the event.
2. the service account running portal and "gis" server for some reason no longer has read/write access to the shared_location
   • are all your machines using the same domain service account? or do they each run a local service account?
   • as you say the process has been running successfully in the past, it would only be an issue if file share permissions have been changed
3. the file server has some hardware issue - bad sectors or a faulty USB connection
   • but if you are seeing the error appear in the webgisdr log at the same place each day, it seems unlikely to be an intermittent hardware problem
   • you may wish check this site for some ideas (I just googled this up)
      https://www.diskgenius.com/how-to/device-attached-to-the-system-is-not-functioning.php 

[DavidHoy]

I agree with jsthusker that using a local share for shared_location and backup_location on the Portal Server where you are running webgisdr is a smart move. Hopefully, you have enough available space.
It will run faster - chiefly because the compress step is slow across a network.  At 10.9.1 the compress library has been changed and is much much faster.