I'm trying to troubleshoot an issue with a staging server on 10.8. When trying to view hosted features layers I get a data error. The browser console lists these as a CORS error but the actual response is {"error":{"code":403,"message":"Forbidden","details":["Request denied due to lack of license."]}}. All licenses should be up to date and what's more if I visit the REST endpoint URL I can browse to it just fine. If I try to query it I get the same error, but if I query it with the explicit Port :6443 added to the Server URL it works just fine. Similarly if I add a hosted feature layer to a web map it doesn't load (or sometimes returns only 1 tile) but if I add it via the REST URL with the explicit port :6443 it loads in the map just fine.
The only thing I could find along these lines is without a web adaptor the REST URLs have that port, but the Portal is successfully configured with the web adaptor and the server is federated. All relevant ports are opened on the Server/Portal/Datastore and I was able to validate the Datastore connection in the Server Manager. It feels like a communication issue with the datastore as everything seems to work fine in the portal until it tries to query information in a hosted feature layer and by default it's not using the explicit port. I'm not sure where to look next.
Edit:
Also checked the web adapter on the ArcGIS server which appears to be configured correctly as well. I also noticed while on the server the same issue happens if I use localhost instead of the webadapter URL. localhost/arcgis/rest/services/Hosted/test_hosted_layer/FeatureServer/0/query gives a different error, {"error":{"code":403,"message":"Forbidden","details":["Request to query a layer has been denied."]}}, but localhost:6443/arcgis/rest/services/Hosted/test_hosted_layer/FeatureServer/0/query works just fine.
Edit2:
One of the weirder things I've noticed is that it'll work once in a while without an explicit port. I'll be sitting on the screen with the error, hit refresh, then I get the query page. Hit refresh immediately again and I'm back to the error. That's at least consistent with the fact that adding it to a web map will sometimes get one successful tile of features displayed. And it will always work when I have the explicit port :6443 in the URL. It doesn't behave this way on the production server and I've gone through as many admin screens as I can looking for differences between the two but haven't found any that have had an effect.
Edit3:
Redid the licensing provisioning file and removed the existing datastore and installed it on the same system to see if it's an inter-machine communication issue, same result. Pretty much any ArcGIS server site works just fine, manager, rest details, any operations other than query, but going to query returns the {"error":{"code":403,"message":"Forbidden","details":["Request denied due to lack of license."]}}. And query works just fine if port :6443 is added to the REST URL.
