topic Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal in ArcGIS Enterprise Questions

Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Wed, 26 May 2021 13:27:32 GMT

Hello - our 10.8.1 upgrade to 10.9 went well. We have a distributed deployment with separate web, portal, hosting and federated sites on Windows 2016 VM machines.

I have re-shared everything I can think of from Living Atlas, Routing, Geocoding services. No issues with our web adaptors, no 'bounce' from DNS alias to machine names, so problems with the web context url. No problem with the 'arcgisonline' redirect uri's. The Monitor Administrator returns valid tokens when testing the portal and the hosting and federated sites, whether using the web adaptor site names or the internal DNS names.

Everything that is supposed to be publically accessible is. However, we are getting an error that I had not seen since 10.6 from the hosting site:

Rest - Exception in get user privileges Server machine 'https://myDNSServerName:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'

Honestly, if I did not have Monitor installed I probably would not have noticed since there is no impact to any of my hosted services that I can tell so far, but the logging is excessive (100s and 1000s of entries) . Very similar to the reply from @danweiss from this old post here:

https://community.esri.com/t5/arcgis-enterprise-portal/arcgis-portal-server-exception-setting-ownership-role/td-p/626444

Any suggestions anyone?

@JonEmch

@JonathanQuinn

@Kathleen_Crombez

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Wed, 26 May 2021 14:13:34 GMT

This behavior reads exactly like this bug from the "Portal for ArcGIS Security 2018 1 Update Patch" :

BUG-000108155 - Endless generateToken requests are triggered in map viewer when token expires for a Portal configured with Integrated Windows Authentication (IWA) and federated with ArcGIS Server.

As we also use IWA Active Directory.

I found another related "Invalid Token" Geonet Post here:

https://community.esri.com/t5/arcgis-enterprise-questions/invalid-token/td-p/598822

Where @JacobDeuel asked:

"I am currently running 10.6.1 and I am getting a bunch of errors related to tokens. Everything still works but it is super slow. Is there a patch like this for version 10.6.1?"

Server machine 'https://URL:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'

@JonathanQuinn replied that

Is that entry logged under DEBUG? If so, it can be ignored. We're working on cleaning up those types of messages.

But like @JacobDeuel , the entries are logged under WARNING.

It is the same REST error and probably will require a security patch as well.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

mstranovsky — Thu, 27 May 2021 18:06:25 GMT

I remember this happening before but i cannot remember what we did to resolve it. Has ESRI responded yet?

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Thu, 27 May 2021 20:59:12 GMT

Not that I'm aware of thanks.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Fri, 28 May 2021 16:15:39 GMT

At this time I think the only thing to do is to go into server admin and increase the log setting to SEVERE. I don't want to have to do that but until anyone else can confirm that they are experiencing the same issue it's all I can do until I can get with tech support.

I've discovered some more 'breaking' changes that have to do with hosted layers and schema changes that I will put into another post.

@JonEmch

@mstranovsky

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Wed, 02 Jun 2021 15:36:36 GMT

So I was able to narrow down this ArcServer error somewhat. What’s happening is this:

WARNING Jun 2, 2021, 10:32:50 AM Exception in get user privileges Server machine 'https://myInternalDNSName:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.' Rest

INFO Jun 2, 2021, 10:32:50 AM Request user: e5928d1a-0e43-4cea-b807-944e56ee7460, Service: System/PublishingToolsEx/GPServer

This Geoprocessing Service: /PublishingToolsEx

Has a bunch of different tasks embedded within it:

Analyze Features for Portal; Available Fonts; Delete Service; Describe Datastore; Download Portal Item Data; ExcelAnalyze; ExcelToTableServer; Export Service; Externalize Service Connections;
Generate Features for Portal; Get Cache Info; Get Database Connection String; Manage Feature Service; Publish Datasets In Data Store; Publish Datasets In Data Stores; Publish Portal Service;
Publish Routing Services; Publish Service Definition;Refresh Service; TablesToExcel; Validate Server Data Store

Where the tasks keep getting called by the GP service, but need a ‘token’ to execute.

So when the task tries to run it throws an ‘Invalid Token’ error in the ArcServer server because it is trying to run the service and execute the task with a user that does not exist:

Request user: e5928d1a-0e43-4cea-b807-944e56ee7460 does not exist in the Portal user store.

I’ll have to get with tech support. They’ll probably just say delete it and re-create it.

@JonEmch

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Wed, 02 Jun 2021 19:12:21 GMT

Actually this:

"Request user: e5928d1a-0e43-4cea-b807-944e56ee7460 does not exist in the Portal user store. "

is incorrect. The "Request user" is not a user but a missing sd file in the arcgisoutput server directory. The ID or name format with the dashes in the string caused me to re-evaluate what that ID could be referring to.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

JonEmch — Thu, 03 Jun 2021 14:01:47 GMT

Hey there David,

I've seen this before (admittedly in older versions of ArcGIS Enterprise), and have reached similar conclusions that it's a bit of a red herring, as mentioned below. What concerns me is that it is happening again at 10.9, and generating excessive amounts of the same message. I would highly recommend logging a support ticket to investigate this.

From your post it seems to be not affecting your environment (volume of messages aside), is that correct?

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

JasonHansel1 — Fri, 04 Jun 2021 11:20:30 GMT

David,

Are you also seeing an increase in the amount of RAM the java processes are consuming? On my end I am seeing a significant increase in RAM being consumed after upgrading from 10.8.1 to 10.9

Thanks,

Jason

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Fri, 04 Jun 2021 15:59:18 GMT

That is correct @JonEmch . Yes agreed re red herring. Yes, agreed re happening again at 10.9. It looks like the last time this occurred it required a portal security patch at 10.6.x

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Fri, 04 Jun 2021 16:04:36 GMT

Hi @JasonHansel1 , no I am not seeing any real memory increases. Monitor is showing both the machines in my hosting site hovering around 23-25 % memory utilization - typical for what I have seen since 10.6 for our environment. If I do I will be sure to let know . . .

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Tue, 08 Jun 2021 14:35:13 GMT

Hi @JonEmch , I went ahead and logged a support case:

You have successfully submitted your support request. Your new case number is: 02824287.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

JonEmch — Tue, 08 Jun 2021 14:47:22 GMT

@DavidColey Noted, thanks!

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

JasonHansel1 — Tue, 08 Jun 2021 14:55:19 GMT

Unregistering and re-registering ArcGIS Datastores has fixed some issues I was seeing with 10.9

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

HamishMills — Tue, 21 Sep 2021 03:57:29 GMT

I am experiencing this same issue after upgrading from 10.8.1 to 10.9. Interestingly we have a test environment that does not replicate this issue. Very interested to see where you get with it.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

rcgarrett — Tue, 21 Sep 2021 21:20:21 GMT

I am experiencing this same issue also after upgrading from 10.8.1 to 10.9. Very interested in a resolution. Hard to separate cause and effect when other problems are occurring. Thanks.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

JRaymond — Wed, 03 Nov 2021 16:07:01 GMT

We are also seeing this issue since upgrading from 10.7.1 to 10.9. It sure makes it messy when reviewing logs to troubleshoot other issues. Like was mentioned, this doesn't seem to affect performance at all, but it's a constant stream of warning messages in the logs.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

Natalie_Runyan — Sat, 20 Nov 2021 02:45:06 GMT

I'm getting same errors on a new 10.9 build (Datastore, Server, Portal on separate machines; federated, hosting; OS Windows 2019). Identity store is SAML Azure AD. Domain service account on all servers have full control to relevant program and content directories.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

JasonHansel1 — Sat, 20 Nov 2021 02:57:26 GMT

You check out the Firewall? I’d also try to unregister the DS and re-register. I have run into a scenario or two where I’ve had to restore the DS.

Re: Enterprise 10.9 token error logging for hosting, federated sites connecting to Portal

DavidColey — Tue, 28 Dec 2021 16:17:45 GMT

Hi @JonEmch - with the 10.9.1 upgrade, the excessive logging no longer occurs.