How to validate Tile Cache data store?

[Jan-Tschada]

We are facing some issues using the Tile Cache data store with ArcGIS Enterprise 10.7.1. We have a base deployment using four separate machines (Portal, Server, Data Store and 2x web adaptor). When we try to publish web Scene/Layers using ArcGIS Pro we receive a

Error - ERROR 001812: Unable to connect to the database used for scene caches (The server took too long to answer. The client has timed out. (status code 28).).)

Publishing feature Services works fine. So it seems to be an issue with the Tile Cache data store only.

Validating using the WebUI of ArcGIS Manager shows an exclamation mark without any details. When using the Admin REST API the validate works fine:

{ "datastore.release": "10.7.1.11595", "datastore.name": "tcs_t3vmfvih", "datastore.isConfigured": "true", "machines": [{ "machine.overallhealth": "Healthy", "datastore.release": "10.7.1.11595", "datastore.release.configstore": "1.4", "platform": "Windows", "machine.isReachable": "true", "hostip": "***.**.**.**", "name": "********.*******.***", "role": "PRIMARY", "dbport": 29081, "healthcheck.enable": "true", "status": "Started", "adminurl": "https://********.*******.***:2443/arcgis/datastoreadmin", "db.isactive": "true", "db.isAccepting": "true" }], "datastore.status": "Started", "datastore.isActiveHA": "true", "datastore.overallhealth": "Healthy", "datastore.isRegistered": "true", "datastore.hasValidServerConnection": "true", "datastore.validServerMachinesList": [{ "machineName": "********.*******.***", "adminURL": "https://********.*******.***:6443/arcgis/admin" }], "owningSystemUrl": "https://********.*******.***:6443/arcgis/admin" }

Only the couchdb log shows some errors:

[error] 2020-01-20T14:45:39.748000Z couchdb@V********.*******.*** <0.28197.147> 78ca4036a3 rexi_server: from: couchdb@********.*******.***(<0.29235.147>) mfa: fabric_rpc:open_shard/2 throw:{unauthorized,<<"You are not authorized to access this db.">>} [{couch_db,open,2,[{file,"src/couch_db.erl"},{line,154}]},{fabric_rpc,open_shard,2,[{file,"src/fabric_rpc.erl"},{line,267}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]

The ports 29080 and 29081 and the erl.exe are allowed to communicate through TCP defined by the Windows Firewall Settings. The datastore is registered using the internal hostname.

Any suggestions how to go further?

[AndrewWhitman1]

Greetings Jan, 

I would only pay attention to the Couchdb message here. There is a db user associated with the tile cache and it cannot authenticate here. Can we run the listmanageduser utility command for this tile cache? If so, I am curious what returns (You do not need to share credentials here obviously) and whether we can log into couchdb with the credentials. 

https://enterprise.arcgis.com/en/data-store/latest/install/windows/data-store-utility-reference.htm#...

Also, what is the OS of these machines? CouchDB uses cookies to authenticate, I am wondering if we might be able to white list some URLs in the internet options if this is Windows. 

The following article touches on what I have in mind for ensuring this authentication request goes through successfully. 

https://support.esri.com/en/Technical-Article/000018751

[StefanUseldinger]

We are also facing the issue under 10.8.1 and are able to reproduce the issue when running the describedatastore tool.

Setting the CouchDb log level to debug clearly shows that the Data Store host is making an anonymous HTTP request which is denied by the db returning the error message and a HTTP 401:

[error] 2020-12-09T09:19:11.617000Z couchdb@*** <0.11580.259> 21fdab6fb7 rexi_server: from: couchdb@***(<0.10514.259>) mfa: fabric_rpc:open_shard/2 throw:{unauthorized,<<"You are not authorized to access this db.">>} [{couch_db,open,2,[{file,"src/couch_db.erl"},{line,154}]},{fabric_rpc,open_shard,2,[{file,"src/fabric_rpc.erl"},{line,267}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]

[notice] 2020-12-09T11:09:08.531000Z couchdb@*** <0.3225.0> ff6486d898 ***:29081 *** undefined GET /dsconfig$/tcs_8qh8d4vz 401 ok 1

We had the same issue, where the WebUI of the ArcGIS Manager didn't validate, but the REST API returned a proper result. After inspecting the Manager network request, I saw that he couldn't find the /validate endpoint. After opening the correct Data Store ports on the Data Store host and the ArcGIS Server machines, followed by logout/login (to avoid token expiry issues) it worked.

However, I am still asking myself, why the describedatastore is producing the error in the CouchDb log and does not show any error in the output of the command line utility. Any idea?

[JulioGarcía]

Hi,

Have anybody find a solution to this? 

We are facing a similar issue (same error on couch log) on our 10.8.1 deployment.

In our case we can´t validate tile cache from ArcGis Server manager and neither from Admin API .

I also have to say, that we can log into CauchDB without problems with both users, the admin one and the managed too.

Thanks in advance.

[StefanUseldinger]

For us, the validation error vanished after opening the ports. Did You validate using WebUI, Rest or Cmd?

I personally think that the CouchDb-logs don't have anything to do with the validation problem. We follow this bug in MyEsri (https://my.esri.com/#/support/bugs/BUG-000130279) and noticed that it still occurs in Enterprise 11.0

[JulioGarcía]

Stefan, Thanks  for reply!

We are trying to validate from WebUI and Rest, in both cases fails.

I read BUG-000130279. The only difference I find with what it describes there, is that I cannot work with the scenes, I cannot publish or use any of the ones that we already have published on the portal

[StefanUseldinger]

Have You opened the ports 2443, 4369, 6443, 9220, 9320, 9876, 29079, 29080, 29081, 29085-29090 or temporary opened the firewall completely for debugging purposes on ArcGIS Server and Data Store?