Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

[DavidColey]

Hello - our 10.8.1 upgrade to 10.9 went well.  We have a distributed deployment with separate web, portal, hosting and federated sites on Windows 2016 VM machines.

I have re-shared everything I can think of from Living Atlas, Routing, Geocoding services.  No issues with our web adaptors, no 'bounce' from DNS alias to machine names, so problems with the web context url.  No problem with the 'arcgisonline' redirect uri's. The Monitor Administrator returns valid tokens when testing the portal and the hosting and federated sites, whether using the web adaptor site names or the internal DNS names.

Everything that is supposed to be publically accessible is.  However, we are getting an error that I had not seen since 10.6 from the hosting site:

 Rest - Exception in get user privileges Server machine 'https://myDNSServerName:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'

Honestly, if I did not have Monitor installed I probably would not have noticed since there is no impact to any of my hosted services that I can tell so far, but the logging is excessive (100s and 1000s of entries) .  Very similar to the reply from @danweiss from this old post here:

https://community.esri.com/t5/arcgis-enterprise-portal/arcgis-portal-server-exception-setting-owners...

Any suggestions anyone?

@JonEmch 

@JonathanQuinn 

@Kathleen_Crombez 

[MichailMarinakis1]

Hi all, I see that this is an old post but recently we came across similar behavior in the logs with our installation (HA enterprise with IWA, upgraded to 10.9.1 some months ago) . 

We are getting warning messages in our federated server e.g.:

Failed to return the service configuration 'X.MapServer'. Could not connect to the ArcGIS component at URL 'https://server.test.com:7443/arcgis/sharing/rest/content/items/xxxxxx'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: Read timed out

The service is working properly, without any issues.

In the portal we also had some weird warnings:

Failed to update the system properties. java.lang.Exception: The server at 'https://portal.test.com:7443/arcgis/sharing/rest/portals/self/servers' returned an error. Invalid token. []

when we tried to update the system properties, even though the properties are properly updated. 

Could it be that we set the portal logs to verbose? Or...

Could it be that we set the portal "disableServicesDirectory": true? Or...

Could it be that some end user is sending requests to some services without the proper headers on their browser?

Just throwing ideas 😀

Everything is working fine overall, it's just our logs are getting "polluted".

Thanks!