Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

[DavidColey]

Hello - our 10.8.1 upgrade to 10.9 went well.  We have a distributed deployment with separate web, portal, hosting and federated sites on Windows 2016 VM machines.

I have re-shared everything I can think of from Living Atlas, Routing, Geocoding services.  No issues with our web adaptors, no 'bounce' from DNS alias to machine names, so problems with the web context url.  No problem with the 'arcgisonline' redirect uri's. The Monitor Administrator returns valid tokens when testing the portal and the hosting and federated sites, whether using the web adaptor site names or the internal DNS names.

Everything that is supposed to be publically accessible is.  However, we are getting an error that I had not seen since 10.6 from the hosting site:

 Rest - Exception in get user privileges Server machine 'https://myDNSServerName:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'

Honestly, if I did not have Monitor installed I probably would not have noticed since there is no impact to any of my hosted services that I can tell so far, but the logging is excessive (100s and 1000s of entries) .  Very similar to the reply from @danweiss from this old post here:

https://community.esri.com/t5/arcgis-enterprise-portal/arcgis-portal-server-exception-setting-owners...

Any suggestions anyone?

@JonEmch 

@JonathanQuinn 

@Kathleen_Crombez 

[DavidColey]

This behavior reads exactly like this bug from the "Portal for ArcGIS Security 2018 1 Update Patch" :

• BUG-000108155 - Endless generateToken requests are triggered in map viewer when token expires for a Portal configured with Integrated Windows Authentication (IWA) and federated with ArcGIS Server.

As we also use IWA Active Directory.

I found another related "Invalid Token" Geonet Post here:

https://community.esri.com/t5/arcgis-enterprise-questions/invalid-token/td-p/598822

Where @JacobDeuel  asked:

"I am currently running 10.6.1 and I am getting a bunch of errors related to tokens.  Everything still works but it is super slow.  Is there a patch like this for version 10.6.1?"

Server machine 'https://URL:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'

@JonathanQuinn replied that 

Is that entry logged under DEBUG? If so, it can be ignored. We're working on cleaning up those types of messages.

But like @JacobDeuel , the entries are logged under WARNING. 

It is the same REST error and probably will require a security patch as well.

[mstranovsky]

I remember this happening before but i cannot remember what we did to resolve it.    Has ESRI responded yet?

[DavidColey]

Not that I'm aware of thanks. 

[DavidColey]

At this time I think the only thing to do is to go into server admin and increase the log setting to SEVERE.  I don't want to have to do that but until anyone else can confirm that they are experiencing the same issue it's all I can do until I can get with tech support. 

I've discovered some more 'breaking' changes that have to do with hosted layers and schema changes that I will put into another post.

@JonEmch 

@mstranovsky 

[DavidColey]

So I was able to narrow down this ArcServer error somewhat.  What’s happening is this:

WARNING           Jun 2, 2021, 10:32:50 AM             Exception in get user privileges Server machine 'https://myInternalDNSName:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'          Rest

INFO      Jun 2, 2021, 10:32:50 AM             Request user: e5928d1a-0e43-4cea-b807-944e56ee7460, Service: System/PublishingToolsEx/GPServer

This Geoprocessing Service: /PublishingToolsEx

Has a bunch of different tasks embedded within it:

Analyze Features for Portal; Available Fonts; Delete Service; Describe Datastore; Download Portal Item Data; ExcelAnalyze; ExcelToTableServer; Export Service; Externalize Service Connections;
Generate Features for Portal; Get Cache Info; Get Database Connection String; Manage Feature Service; Publish Datasets In Data Store; Publish Datasets In Data Stores; Publish Portal Service;
Publish Routing Services; Publish Service Definition;Refresh Service; TablesToExcel; Validate Server Data Store

Where the tasks keep getting called by the GP service, but need a ‘token’ to execute. 

So when the task tries to run it throws an ‘Invalid Token’ error in the ArcServer server because it is trying to run the service and execute the task with a user that does not exist:

Request user: e5928d1a-0e43-4cea-b807-944e56ee7460 does not exist in the Portal user store.  

I’ll have to get with tech support.  They’ll probably just say delete it and re-create it.

@JonEmch 

[DavidColey]

Actually this:

"Request user: e5928d1a-0e43-4cea-b807-944e56ee7460 does not exist in the Portal user store. "

is incorrect.  The "Request user" is not a user but a missing sd file in the arcgisoutput server directory.  The ID or name format with the dashes in the string caused me to re-evaluate what that ID could be referring to.

[JonEmch]

Hey there David,

I've seen this before (admittedly in older versions of ArcGIS Enterprise), and have reached similar conclusions that it's a bit of a red herring, as mentioned below. What concerns me is that it is happening again at 10.9, and generating excessive amounts of the same message. I would highly recommend logging a support ticket to investigate this.

From your post it seems to be not affecting your environment (volume of messages aside), is that correct?

[DavidColey]

That is correct @JonEmch . Yes agreed re red herring.  Yes, agreed re happening again at 10.9.  It looks like the last time this occurred it required a portal security patch at 10.6.x

[DavidColey]

Hi @JonEmch , I went ahead and logged a support case:

You have successfully submitted your support request. Your new case number is: 02824287.