ArcGIS Server Service randomly returns a "Invalid Token"

[CarstenHogertz]

I have the problem that an arcGIS Server service has recently started generating warnings in the log. The content of the warnings is as follows:

Exception in get user privileges Failed to return the service configuration 'ServiceName.MapServer'. Could not connect to the ArcGIS component at URL 'https://webadaptorurl/portal/sharing/rest/content/items/b0a3f18992264ca2b'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: Read timed out

Failed to return the service configuration 'ServiceName.MapServer'. Could not connect to the ArcGIS component at URL 'https://webadaptorurl/portal/sharing/rest/content/items/b0a3f18992264ca2b'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: Read timed out

The service is called in a web application.
This is a secured service. The user of the web application is logged into the portal.
The service itself has approx. 40 sublayers. Now the web application requests the individual sublayers including tokens. The token is correct and is also correctly attached to each request. The strange thing, however, is that some sublayers report back that the token is invalid. These are not always the same sublayers, but the REST interface reports this seemingly randomly to some sublayers.

The whole construct has worked reliably for years. It is not possible to pinpoint why the problem suddenly occurs. I have also not found any help in relation to ESRI products.
One reference, but not from the GIS environment, says it could have something to do with CORS or RequestHeaders.
I have no idea where to start troubleshooting.
Thank you very much,
Carsten

[MarcoBoeringa]

Since the error states a "time out", has anything changed lately in your configuration to bog down your server, and maybe cause such a time out?

Since you already run this for years, possibly on the same hardware, your hardware may no longer be able to fully keep up, and cause time out on some requests. Certainly, if your database and usage has grown considerably over the years, maybe the server is running against its limits. Have you checked that?

[CarstenHogertz]

These two logs are ArcGIS Server Logs. 

The browser's DevTools show in the response tab of some sublayers "invalid token"

[DanMori]

Hi.  We are experiencing the same issue.  This error happens intermittently.  A call to a service fails with the reported error message, yet the same call to the same service later succeeds.  

We've had this since 10.8 enterprise.  The issue still remains after an upgrade to 10.9.1.

Looks to be a communication issue between ArcServer and Portal.  The first thing it is trying to do is to validate the token.  Which seems to fail due to a lack of communication.  

@CarstenHogertz Do you have the link about a possible CORS problem..

[JoshuaBixby]

Without knowing more specifics about how ArcGIS Enterprise is deployed, it is hard for community members to provide feedback.  For example, are you running ArcGIS Enterprise on-premises or in the cloud?  If in the cloud, did you deploy it using one of Esri's tools or deploy it yourself?  Are all the components on the same machine or do you have separate machines for each component?  Are you running High Availability portal?  Are you running multiple-machine ArcGIS Server sites? Etc.....

ArcGIS Enterprise isn't a piece of software but a system made up of many pieces of software, and how those individual pieces are deployed can very much influence troubleshooting.