nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey

Hello - our 10.8.1 upgrade to 10.9 went well. We have a distributed deployment with separate web, portal, hosting and federated sites on Windows 2016 VM machines.

I have re-shared everything I can think of from Living Atlas, Routing, Geocoding services. No issues with our web adaptors, no 'bounce' from DNS alias to machine names, so problems with the web context url. No problem with the 'arcgisonline' redirect uri's. The Monitor Administrator returns valid tokens when testing the portal and the hosting and federated sites, whether using the web adaptor site names or the internal DNS names.

Everything that is supposed to be publically accessible is. However, we are getting an error that I had not seen since 10.6 from the hosting site:

 Rest - Exception in get user privileges Server machine 'https://myDNSServerName:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'

Honestly, if I did not have Monitor installed I probably would not have noticed since there is no impact to any of my hosted services that I can tell so far, but the logging is excessive (100s and 1000s of entries) . Very similar to the reply from @danweiss from this old post here:

https://community.esri.com/t5/arcgis-enterprise-portal/arcgis-portal-server-exception-setting-ownership-role/td-p/626444

Any suggestions anyone?

@JonEmch

@JonathanQuinn

@Kathleen_Crombez

Find more posts tagged with

Accepted answers

DavidColey

Hi @JonEmch - with the 10.9.1 upgrade, the excessive logging no longer occurs.

All comments

DavidColey

This behavior reads exactly like this bug from the "Portal for ArcGIS Security 2018 1 Update Patch" :

BUG-000108155 - Endless generateToken requests are triggered in map viewer when token expires for a Portal configured with Integrated Windows Authentication (IWA) and federated with ArcGIS Server.

As we also use IWA Active Directory.

I found another related "Invalid Token" Geonet Post here:

https://community.esri.com/t5/arcgis-enterprise-questions/invalid-token/td-p/598822

Where @JacobDeuel asked:

"I am currently running 10.6.1 and I am getting a bunch of errors related to tokens. Everything still works but it is super slow. Is there a patch like this for version 10.6.1?"

Server machine 'https://URL:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'

@JonathanQuinn replied that

Is that entry logged under DEBUG? If so, it can be ignored. We're working on cleaning up those types of messages.

But like @JacobDeuel , the entries are logged under WARNING.

It is the same REST error and probably will require a security patch as well.

mstranovsky

I remember this happening before but i cannot remember what we did to resolve it. Has ESRI responded yet?

DavidColey

Not that I'm aware of thanks.

DavidColey

At this time I think the only thing to do is to go into server admin and increase the log setting to SEVERE. I don't want to have to do that but until anyone else can confirm that they are experiencing the same issue it's all I can do until I can get with tech support.

I've discovered some more 'breaking' changes that have to do with hosted layers and schema changes that I will put into another post.

@JonEmch

@mstranovsky

DavidColey

So I was able to narrow down this ArcServer error somewhat. What’s happening is this:

WARNING Jun 2, 2021, 10:32:50 AM Exception in get user privileges Server machine 'https://myInternalDNSName:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.' Rest

INFO Jun 2, 2021, 10:32:50 AM Request user: e5928d1a-0e43-4cea-b807-944e56ee7460, Service: System/PublishingToolsEx/GPServer

This Geoprocessing Service: /PublishingToolsEx

Has a bunch of different tasks embedded within it:

Analyze Features for Portal; Available Fonts; Delete Service; Describe Datastore; Download Portal Item Data; ExcelAnalyze; ExcelToTableServer; Export Service; Externalize Service Connections;
Generate Features for Portal; Get Cache Info; Get Database Connection String; Manage Feature Service; Publish Datasets In Data Store; Publish Datasets In Data Stores; Publish Portal Service;
Publish Routing Services; Publish Service Definition;Refresh Service; TablesToExcel; Validate Server Data Store

Where the tasks keep getting called by the GP service, but need a ‘token’ to execute.

So when the task tries to run it throws an ‘Invalid Token’ error in the ArcServer server because it is trying to run the service and execute the task with a user that does not exist:

Request user: e5928d1a-0e43-4cea-b807-944e56ee7460 does not exist in the Portal user store.

I’ll have to get with tech support. They’ll probably just say delete it and re-create it.

@JonEmch

DavidColey

Actually this:

"Request user: e5928d1a-0e43-4cea-b807-944e56ee7460 does not exist in the Portal user store. "

is incorrect. The "Request user" is not a user but a missing sd file in the arcgisoutput server directory. The ID or name format with the dashes in the string caused me to re-evaluate what that ID could be referring to.

Jon Emch

Hey there David,

I've seen this before (admittedly in older versions of ArcGIS Enterprise), and have reached similar conclusions that it's a bit of a red herring, as mentioned below. What concerns me is that it is happening again at 10.9, and generating excessive amounts of the same message. I would highly recommend logging a support ticket to investigate this.

From your post it seems to be not affecting your environment (volume of messages aside), is that correct?

JasonHansel1

David,

Are you also seeing an increase in the amount of RAM the java processes are consuming? On my end I am seeing a significant increase in RAM being consumed after upgrading from 10.8.1 to 10.9

Thanks,

Jason

DavidColey

That is correct @JonEmch . Yes agreed re red herring. Yes, agreed re happening again at 10.9. It looks like the last time this occurred it required a portal security patch at 10.6.x

DavidColey

Hi @JasonHansel1 , no I am not seeing any real memory increases. Monitor is showing both the machines in my hosting site hovering around 23-25 % memory utilization - typical for what I have seen since 10.6 for our environment. If I do I will be sure to let know . . .

DavidColey

Hi @JonEmch , I went ahead and logged a support case:

You have successfully submitted your support request. Your new case number is: 02824287.

Jon Emch

@DavidColey Noted, thanks!

JasonHansel1

Unregistering and re-registering ArcGIS Datastores has fixed some issues I was seeing with 10.9

HamishMills

I am experiencing this same issue after upgrading from 10.8.1 to 10.9. Interestingly we have a test environment that does not replicate this issue. Very interested to see where you get with it.

RayGarrett

I am experiencing this same issue also after upgrading from 10.8.1 to 10.9. Very interested in a resolution. Hard to separate cause and effect when other problems are occurring. Thanks.

JRaymond

We are also seeing this issue since upgrading from 10.7.1 to 10.9. It sure makes it messy when reviewing logs to troubleshoot other issues. Like was mentioned, this doesn't seem to affect performance at all, but it's a constant stream of warning messages in the logs.

Natalie_Runyan

I'm getting same errors on a new 10.9 build (Datastore, Server, Portal on separate machines; federated, hosting; OS Windows 2019). Identity store is SAML Azure AD. Domain service account on all servers have full control to relevant program and content directories.

JasonHansel1

You check out the Firewall? I’d also try to unregister the DS and re-register. I have run into a scenario or two where I’ve had to restore the DS.

MichailMarinakis1

Hi all, I see that this is an old post but recently we came across similar behavior in the logs with our installation (HA enterprise with IWA, upgraded to 10.9.1 some months ago) .

We are getting warning messages in our federated server e.g.:

Failed to return the service configuration 'X.MapServer'. Could not connect to the ArcGIS component at URL 'https://server.test.com:7443/arcgis/sharing/rest/content/items/xyz'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: Read timed out

The service is working properly, without any issues.

In the portal we also had some weird warnings:

Failed to update the system properties. java.lang.Exception: The server at 'https://portal.test.com:7443/arcgis/sharing/rest/portals/self/servers' returned an error. Invalid token. []

when we tried to update the system properties, even though the properties are properly updated.

Could it be that we set the portal logs to verbose? Or...

Could it be that we set the portal "disableServicesDirectory": true? Or...

Could it be that some end user is sending requests to some services without the proper headers on their browser?

Just throwing ideas 😀

Everything is working fine overall, it's just our logs are getting "polluted".

Thanks!

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.