Backup TileCache Datastore failed caused by Timeout error on tile cache data store.

[Suleyman-Arslan]

Hello,

I get below error when I try to backup tilecache datastore. I could not find this error on community. No body face this before? 

>backupdatastore backuptilecachetest --store tileCache --location type=fs;name=network
You are attempting to back up the data store. This process could take a long time, depending on the size of your data. Please do not interrupt the process once it has started.

Do you want to continue (Yes or No)?y

Error encountered: Machine 'https://xxx:2443/arcgis/datastoreadmin' returned an error. 'Backup on data store machine 'xxx' failed.
Caused by: Timeout error on tile cache data store.'

Datastore configuration is:

Information for tile cache data store tcs_gsh9mezg
==================================================
Data location.......................P:\ArcGISDataStore\nosqldata
Data store status...................Started
Backup location.....................//......../datastore/tilecache/
Backup schedule.....................{"schedule-starttime":"23:00:00","schedule-frequency":"Every 5 DAYS"}
Mode................................primaryStandby
member tile cache machines..........host1.com host2.com
Owning system URL...................https://server1.com:6443/arcgis/admin
Portal for ArcGIS URL...............https://portal.com/portal

[Scott_Tansley]

Is the P: local to the machine?  How much data is there?  Where's the export being written to, where's that server, what disk does it have?

Sorry, lots of questions, but as an example,  I had a similar issue recently.  The client had provided servers that were SSD in an IaaS datacentre, but there were limited IOPs available (per hour).  The backup process was using the avialable read/write capacity within 10 minutes, but needed 35 minutes to create the backup.  

The fact that you're timing out suggests that it cannot commit the data fast enough....  Bear in mind that tile cache can be pretty big.

The temporary work around for us was to ensure that the backup was 'local' to the datastore machine.  Then a scheduled task copied the backup to a file server as a secondary (later) step.

[Suleyman-Arslan]

Hello @Scott_Tansley 

Thank you for your quick reply.

The P: is local drive and contains 7.29 GB of data.

The backup location is network file share.

The virtual servers are located on customers on premise data center. And I don't think there is an IOPs limitation

Timeout error happened within about two minute. 

Actually I get this error during webgisdr export. then try manually backup.

I can create webgisdr backup successfully without tilecache, but it fails on tilecache backup if I include tilecache in webgisdr.

[Scott_Tansley]

That's all good info.  Can I ask if the tile cache is actually used, or are you just being belt and braces.  Very few of my clients are actually using Scenes, and therefore backing up the tile cache datastore (for most) is irrelevant?

For that reason, I may not be able to go much further.  

I suppose one question to ask is, does the service account that that the ArcGIS Datastore runs under have permissions to write to the Network File Share?  The speed of failure suggests it could be trying to connect and being rejected? 

[Suleyman-Arslan]

Yes we have scenes stored in tilcecache datastore sized over 7 gb.

And I confirm that service account has write access to this location. I am creating webgisdr exports to the same location without any error.