ArcGIS Enterprise 11.1-11.2: Restoring tile cache breaks scene layers

[NicolasGIS]

Hello,

I am facing an issue since we upgraded from 11.0 to 11.1: when restoring our tile cache datastore using webgisdr on standby environment, scene layers restored are corrupted and broken.

Though the restore is said to be successful, when trying to visualize them, it fails.
All requests querying for tiles description of the service fail:
=> Hosted/Buildings_3D/SceneServer/layers/0/nodes/root?f=json
returns:

{
    "error": {
        "code": 500,
        "message": "Server error unable to process request",
        "details": []
    }
}

But validation of the datastores from ArcGIS Server Manager or ArcGIS Server admin interface is successful. Webgisdr log say it successful as well.

On ArcGIS Server Manager in debug mode, this 500 errors seems to be related to 404 from CouchDB:

This issue occurs on both ArcGIS Enterprise deployments we are running. The procedure to restore on standby is implemented since several versions and we never faced any issue (except at 11.0 with restoring relational datastore cf. https://community.esri.com/t5/high-availability-and-disaster-recovery-questions/cannot-restore-arcgi... that was fixed later with a patch).

I opened a case 6 months ago after the upgrade with support (case 03338742) and so far no progress...
What we tried:
- manually restoring the tile cache datastore from production tile cache backup: same results
- manually restoring the tile cache datastore from webgisdr backup: same results
- restoring on a datastore content folder stored on C:\ of the VM to exclude drive performance issue: same results

Interesting facts:
- It's not systematic - sometimes all the scene services are broken, sometimes only some of them but never all of them were successfully restored. But most of the time, they are all broken.
- Procedure repeated tens of times with always the same result
- Publishing a new scene layer on the standby datastore with broken restored services works: the tile cache datastore is functional
- Publishing a new scene layer on the production tile cache datastore and restoring it fails as well (it is not restriceted to all scene layers published from previous versions)

Worth to mention, that we could potentially loose all our work as there is currently no way to export scene layers as a backup solution because of "BUG-000143562 - Exporting a multipatch hosted feature layer as a file geodatabase from ArcGIS Online results in a corrupted geodatabase" so there is currently no way to retrieve tile cache datastore content...

I would really appreciate your inputs @ChristopherPawlyszyn and @JonathanQuinn ...

Thanks

[ChristopherPawlyszyn]

@NicolasGIS - I'll need a bit to look into reproducing this, we're wrapped up in certification for our next release right now; will touch base with Support contacts as well. Hopefully I can get to some testing later this week and I'll report back.

[NicolasGIS]

Hello @ChristopherPawlyszyn,

I continue my investigations and wanted to report the following facts: 

- After authenticating on couchDB, I can see that all our nosql "databases" are empty (size 0 bytes) and made of 0 documents: thus the 404 couchDB error.

- I activated couchDB debug log when doing the webgisdr restore and it turns out I have plenty of errors 409 conflict like the following:

[debug] 2023-10-26T13:18:36.924000Z couchdb@MYHOST.COMPANY.COM <0.2262.0> 7e7d7f7d69 'PUT' /_dbs/ext_site_callout_0_d0d73bef0979452a9660097f2df37359 {1,1} from 'couchdb@MYHOST.COMPANY.COM'
Headers: [{'Accept-Encoding',"gzip,deflate"},{'Connection',"Keep-Alive"},{'Content-Length',"187"},{'Content-Type',"application/json"},{'Cookie',"AuthSession=ItsASecret"},{'Host',"MYHOST.COMPANY.COM:29080"},{'User-Agent',"Apache-HttpClient/4.5.13 (Java/11.0.18)"}]
[debug] 2023-10-26T13:18:36.925000Z couchdb@MYHOST.COMPANY.COM <0.2262.0> 7e7d7f7d69 timeout 600
[debug] 2023-10-26T13:18:36.925000Z couchdb@MYHOST.COMPANY.COM <0.2262.0> 7e7d7f7d69 Successful cookie auth as: "admin_u92a8"
[debug] 2023-10-26T13:18:36.926000Z couchdb@MYHOST.COMPANY.COM <0.2262.0> 7e7d7f7d69 Minor error in HTTP request: conflict
[debug] 2023-10-26T13:18:36.926000Z couchdb@MYHOST.COMPANY.COM <0.2262.0> 7e7d7f7d69 Stacktrace: [{couch_db,update_doc,4,[{file,"src/couch_db.erl"},{line,839}]},{couch_httpd_db,update_doc,6,[{file,"src/couch_httpd_db.erl"},{line,780}]},{couch_httpd_db,do_db_req,2,[{file,"src/couch_httpd_db.erl"},{line,261}]},{couch_httpd,handle_request_int,5,[{file,"src/couch_httpd.erl"},{line,339}]},{erpc,execute_call,3,[{file,"erpc.erl"},{line,434}]},{erpc,call,5,[{file,"erpc.erl"},{line,117}]},{rpc,call,5,[{file,"rpc.erl"},{line,410}]},{chttpd_node,call_node,4,[{file,"src/chttpd_node.erl"},{line,218}]}]

Looking at couchDB documentation:

https://docs.couchdb.org/en/stable/api/document/attachments.html#put--db-docid-attname

409 Conflict – Document’s revision wasn’t specified or it’s not the latest

 Any progress on your side ? 

Thanks,

Nicolas

[ChristopherPawlyszyn]

Thanks for the additional information @NicolasGIS, we're still looking into this more internally but it appears there is a defect in the new backup/restore logic implemented for the tile cache data store at 11.1 for scene services published at an earlier version. I'm working with Support to get a bug logged in the system.

We can set the data store to use the previous method, as this was successful in my testing. The following changes should be performed on all ArcGIS Data Store machines with the tile cache data store configured:

1. Modify the tilecache_backup_type property in <ArcGIS Data Store install path>/framework/etc/datastore.properties to REPLICATION_BACKUP
2. Restart the ArcGIS Data Store service
3. Take a new backup from the primary deployment

Once the changes are complete for the machines in the site:

1. Unregister the existing backup location (if configured) and delete the contents on the secondary deployment, then re-register the same location
   
   1. If using the DEFAULT_LOCATION (<arcgisdatastore>/backup/tilecache), will need to register a new backup location and set as default
2. Restore to the secondary deployment
3. Verify scene layers are accessible on the secondary deployment

[NicolasGIS]

Many thanks @ChristopherPawlyszyn  for your reply.

there is a defect in the new backup/restore logic implemented for the tile cache data store at 11.1 for scene services published at an earlier version

In case that help, as mentionned in my first post, in my case, the problem also occurs for sceneLayers published directly at 11.1:

Publishing a new scene layer on the production tile cache datastore and restoring it fails as well (it is not restriceted to all scene layers published from previous versions)

I just tried your workaround on one of our ArcGIS Enterprise deployment, and the issue remains. I tried updating the datastore configuration on the standby environment as well , but it did not change the result.

I tried inspecting files in the webgisdr backup, and tile cache datastores look pretty similar. How can I tell whether webgisdr was exported with "REPLICATION_BACKUP" or "DIRECT_FILE_BACKUP" ?

Thanks

[ChristopherPawlyszyn]

@NicolasGIS I'm still working through some iterations, I had tried on two separate systems so had one that hadn't gone through the backup process already. Will report back.

[ChristopherPawlyszyn]

@NicolasGIS I updated the path and steps above for clarity as well as some information about the backup location, the backup type is pulled from the installation location and not the site-specific location.

In terms of verifying the backup/restore method used, there is a backup metadata file included in the compressed webgissite file that contains that information.

[NicolasGIS]

Thanks for your reply @ChristopherPawlyszyn.

I modified the correct file (I was indeed using from the site-specific location), and tried 4 times: always the same error:

{
    "jobId": "fcfdaa47-d8fd-4993-b292-69cdd3386abc",
    "errorMessage": "Failed to register tile cache data store.\nCaused by: The specified GIS Server site already has a tile cache data store.",
    "description": "Deploy data store snapshot 20231101-210033-97-BACKUP from \\\\pathto\\WebGISSite1698922766136\\dataStore\\be7a1844-1f26-4faa-97b9-5b69dd2dec6f",
    "lastModified": "2023-11-02 12:06",
    "status": "failed"
}

From a datastore server log:

<Msg time="2023-11-01T22:01:19,873" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Successfully restored and registered Primary relational machine...</Msg>
<Msg time="2023-11-01T22:01:20,820" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Successfully applied WebGIS DR snapshot for relational data store...</Msg>
<Msg time="2023-11-01T22:01:29,226" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Deploying files from WebGIS DR snapshot for tile cache data store...</Msg>
<Msg time="2023-11-01T22:01:56,407" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Applying WebGIS DR snapshot for tile cache data store...</Msg>
<Msg time="2023-11-01T22:01:56,463" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Performing restore on a existing tile cache data store...</Msg>
<Msg time="2023-11-01T22:01:56,484" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Found backup set specified:DEFAULT_LOCAL</Msg>
<Msg time="2023-11-01T22:01:56,484" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Validate storage store for the source backup store...</Msg>
<Msg time="2023-11-01T22:01:56,484" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Validate source backup set...</Msg>
<Msg time="2023-11-01T22:01:56,546" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">prepare for tile cache data store configuration...</Msg>
<Msg time="2023-11-01T22:01:56,546" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Configure vm.args...</Msg>
<Msg time="2023-11-01T22:01:56,550" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Prepare default.ini...</Msg>
<Msg time="2023-11-01T22:01:56,552" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Prepare SSL certificates...</Msg>
<Msg time="2023-11-01T22:01:56,736" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Configure C:\arcgisdatastore\staging\tileCache\tcs_dyv65252\etc\local.ini...</Msg>
<Msg time="2023-11-01T22:02:32,159" type="WARNING" code="110379" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Failed to restore 'tile cache' data store. null</Msg>
<Msg time="2023-11-01T22:02:32,159" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">com.esri.arcgis.datastore.common.DataStoreException
at com.esri.arcgis.datastore.model.couchdb.CouchDBManager.restore(CouchDBManager.java:8270)
at com.esri.arcgis.datastore.client.DataStoreAdminClient.b(DataStoreAdminClient.java:4875)
at com.esri.arcgis.datastore.client.DataStoreAdminClient.restore(DataStoreAdminClient.java:3814)
at com.esri.arcgis.datastore.client.DataStoreAdminClient.a(DataStoreAdminClient.java:12847)
at com.esri.arcgis.datastore.client.DataStoreAdminClient.deployWebGISSnapshot(DataStoreAdminClient.java:11789)
at com.esri.arcgis.datastore.client.DataStoreAdminClient$e.run(DataStoreAdminClient$e.java:10924)
at java.base/java.lang.Thread.run(Unknown Source)
</Msg>
<Msg time="2023-11-01T22:02:32,159" type="SEVERE" code="111002" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Failed to import data to your replicated site. Failed to restore 'tile cache' data store.
Caused by: null</Msg>
<Msg time="2023-11-01T22:02:32,160" type="DEBUG" code="9999" source="Data Store" process="24840" thread="19" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">com.esri.arcgis.datastore.common.DataStoreException: Failed to restore 'tile cache' data store.
Caused by: null
at com.esri.arcgis.datastore.client.DataStoreAdminClient.restore(DataStoreAdminClient.java:3819)
at com.esri.arcgis.datastore.client.DataStoreAdminClient.a(DataStoreAdminClient.java:12847)
at com.esri.arcgis.datastore.client.DataStoreAdminClient.deployWebGISSnapshot(DataStoreAdminClient.java:11789)
at com.esri.arcgis.datastore.client.DataStoreAdminClient$e.run(DataStoreAdminClient$e.java:10924)
at java.base/java.lang.Thread.run(Unknown Source)
</Msg>

Any idea ?

Thanks,

Nicolas

[ChristopherPawlyszyn]

@NicolasGIS just to be sure, is this when using the restoredatastore tool included with ArcGIS Data Store or the WebGISDR tool included with Portal for ArcGIS? If the former, you'll need to unregister the data store item from Server Admin (or Manager) before performing the restore.

[NicolasGIS]

@ChristopherPawlyszyn Thanks for your reply.

This is from a webgisdr restore.

I just tried once again. Here is the full error stack from datastore server log in case that help:

<Msg time="2023-11-07T23:57:22,616" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Applying WebGIS DR snapshot for tile cache data store...</Msg>
<Msg time="2023-11-07T23:57:24,878" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Stopping nonsql database </Msg>
<Msg time="2023-11-07T23:57:24,880" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">NoSQL database was stopped </Msg>
<Msg time="2023-11-07T23:57:31,995" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">prepare for tile cache data store configuration...</Msg>
<Msg time="2023-11-07T23:57:31,995" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Configure vm.args...</Msg>
<Msg time="2023-11-07T23:57:31,999" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Prepare default.ini...</Msg>
<Msg time="2023-11-07T23:57:32,1" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Prepare SSL certificates...</Msg>
<Msg time="2023-11-07T23:57:32,160" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Configure C:\arcgisdatastore\nosqldata\etc\local.ini...</Msg>
<Msg time="2023-11-07T23:57:33,167" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">configuring tile cache data store...</Msg>
<Msg time="2023-11-07T23:57:35,653" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">create user usr_t9v56...</Msg>
<Msg time="2023-11-07T23:57:36,681" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">create tile cache site store if needed ...</Msg>
<Msg time="2023-11-07T23:57:36,791" type="WARNING" code="110326" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Backup location '{0}' is not owned by the current data store. Backup location 'C:\arcgisdatastore\backup\tilecache' is already owned by another data store 'tcs_ksvff8l8'.</Msg>
<Msg time="2023-11-07T23:57:36,791" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">com.esri.arcgis.datastore.common.DataStoreException: Backup location 'C:\arcgisdatastore\backup\tilecache' is already owned by another data store 'tcs_ksvff8l8'.
	at com.esri.arcgis.datastore.model.backup.BackupSignature.signBackupFolder(BackupSignature.java:85)
	at com.esri.arcgis.datastore.model.couchdb.CouchDBManager.configure(CouchDBManager.java:497)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient.configCouchDB(DataStoreAdminClient.java:13915)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient.a(DataStoreAdminClient.java:12721)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient.deployWebGISSnapshot(DataStoreAdminClient.java:11789)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient$e.run(DataStoreAdminClient$e.java:10924)
	at java.base/java.lang.Thread.run(Unknown Source)
</Msg>
<Msg time="2023-11-07T23:57:39,101" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Register new title cache data store...</Msg>
<Msg time="2023-11-07T23:57:47,820" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Register a default backup location for tile cache data store...</Msg>
<Msg time="2023-11-07T23:57:47,820" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">New Backup Location will be: E:/backup/tileCache/</Msg>
<Msg time="2023-11-07T23:57:47,820" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">creating backup store for tile cache data store....</Msg>
<Msg time="2023-11-07T23:57:47,833" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">prepare file based repository provider for tile cache data store...</Msg>
<Msg time="2023-11-07T23:57:47,834" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">prepare file based repository provider for tile cache data store on local machine...</Msg>
<Msg time="2023-11-07T23:57:47,868" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">configuring repository for  the  backup store...</Msg>
<Msg time="2023-11-07T23:57:47,893" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Digital sign the backup store...</Msg>
<Msg time="2023-11-07T23:57:48,253" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">prepare for tile cache data store configuration...</Msg>
<Msg time="2023-11-07T23:57:48,254" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Configure vm.args...</Msg>
<Msg time="2023-11-07T23:57:48,257" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Prepare default.ini...</Msg>
<Msg time="2023-11-07T23:57:48,259" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Prepare SSL certificates...</Msg>
<Msg time="2023-11-07T23:57:48,478" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Configure C:\arcgisdatastore\staging\tileCache\tcs_dyv65252\etc\local.ini...</Msg>
<Msg time="2023-11-07T23:58:05,243" type="DEBUG" code="110616" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Webhooks database for site 'd3e74a59-4154-43a2-a5a9-cc0f5dcfdf4b' is not created and registered.</Msg>
<Msg time="2023-11-07T23:58:50,951" type="SEVERE" code="111002" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Failed to import data to your replicated site. null</Msg>
<Msg time="2023-11-07T23:58:50,951" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">com.esri.arcgis.datastore.common.DataStoreException
	at com.esri.arcgis.datastore.model.couchdb.CouchDBManager.restore(CouchDBManager.java:8270)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient.a(DataStoreAdminClient.java:12934)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient.deployWebGISSnapshot(DataStoreAdminClient.java:11789)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient$e.run(DataStoreAdminClient$e.java:10924)
	at java.base/java.lang.Thread.run(Unknown Source)
</Msg>
<Msg time="2023-11-07T23:58:50,952" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Stopping nonsql database </Msg>
<Msg time="2023-11-07T23:58:50,954" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">NoSQL database was stopped </Msg>
<Msg time="2023-11-07T23:58:53,161" type="WARNING" code="110804" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">Failed to register tile cache data store. The specified GIS Server site already has a tile cache data store.</Msg>
<Msg time="2023-11-07T23:58:53,161" type="DEBUG" code="9999" source="Data Store" process="9940" thread="25" methodName="" machine="MAPS11.COMPANY.COM" user="" elapsed="" requestID="">com.esri.arcgis.datastore.common.DataStoreException: The specified GIS Server site already has a tile cache data store.
	at com.esri.arcgis.datastore.model.couchdb.CouchDBManager.registerDataStore(CouchDBManager.java:661)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient.registerTileCacheDataStore(DataStoreAdminClient.java:13937)
	at com.esri.arcgis.datastore.client.DataStoreAdminClientForServer.registerDataStore(DataStoreAdminClientForServer.java:2856)
	at com.esri.arcgis.datastore.client.DataStoreAdminClientForServer.registerDataStore(DataStoreAdminClientForServer.java:2757)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient.a(DataStoreAdminClient.java:13044)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient.deployWebGISSnapshot(DataStoreAdminClient.java:11789)
	at com.esri.arcgis.datastore.client.DataStoreAdminClient$e.run(DataStoreAdminClient$e.java:10924)
	at java.base/java.lang.Thread.run(Unknown Source)
</Msg>

In couch log, no error though.