ArcGIS Data Store "Failed to log in" error message

[FraserHand]

Hi there,

We are having the same issue as described here

https://community.esri.com/t5/arcgis-enterprise-questions/keep-getting-a-warning-in-server-log-says-...

When validating the data stores via the admin API I see the same token error coming from data store

Validate: Machine 'https://...:6443/arcgis/admin/generateToken' returned an error. 'Failed to log in. Invalid username or password specified.'

then the 

failed to log in. Invalid username 'some guid' or password specified.

The only place I've seen any sort of credentials stored is in the dsadmindb in the <data store schema> storeregistry table. There are the following columns auth,storekey, storepwd, sharedkey and regtarget that store some encrypted data - are these related back to Server and are we able to reset this auth if it is the auth that the data store uses to try to generate the token. In the other post @JonathanQuinn says

"Data Store shouldn't store the PSA credentials as Server supports those credentials changing. It should establish a trust between itself and the Server."

How is this trust established and can we reset it without having to unregister the data store - which seems to be the common fix in the thread.

Thanks

Fraser

@MarcGraham2 

[Scott_Tansley]

Hey Fraser.  I see this on approximately 50% of my clients.  It's where the PSA has been disabled in the Hosting Server, and the Data Store is doing a 5 minute ping.  I don't get it as the Data Store is supposed to have a long running (semi-permanent) token and shouldn't need those credentials.  It does not seem to affect performance or reliability of the Hosting Server, it just fills the logs with SPAM.  

I have one client who has equally configured environments for prod and pre-prod, one environment gets the messages the other doesn't.  

I've also seen it randomly stop creating errors in one client's environment.

#Weird

[FraserHand]

Thanks Scott,

Marc gets this as well and neither of us have disabled the PSA. Even if you reset the PSA password it isn't propagated to the data store. When you validate the data store from the Admin API data store logs the fact it is trying to generate a new token but the credentials it has aren't correct. This suggests that these are cached somewhere in data store when the data store is registered with server. I have detailed above where I found some credentially type stuff. Even if DS had a token and it was semi permanent, changing the PSA password would invalidate it right so it would still need to generate a new one - so the password change would need to flow through to ds?

Thanks

[MarcGraham2]

Hi @FraserHand and @Scott_Tansley ,

Last night we ran the unregisterdatastore command on the Tile Cache Data Store, and then the registerdatastore command with the most up to date Primary Site Administrator credentials.

This was successful in resolving the 'Failed to log in. Invalid username or password specified.' error that we were seeing every 5 minutes.  So it seems to be proof that the credentials were cached in the data store config somewhere, and running these tools was successful in updating the creds with no loss of scene layer functionality in Portal.

Cheers,

Marc

[Scott_Tansley]

Good sleuthing.  

[MarcGraham2]

@Scott_Tansley - It was your idea!! 😂

Thanks for the help.

[FraserHand]

There are a few fields in the dbadmin table which have what looks like encrypted data

auth, storekey, storepwd, sharedkey

and I suspect the creds are in here somewhere (it's just a hunch) - if the pw changes on server and is swapped back maybe there is a shared key update or something which invalidates what is stored in ds, giving the invalid user / pw. Again - just a hunch.

Fraser

[Scott_Tansley]

I'm perplexed by this whole issue.  We set up a PSA on the hosting server, stored the creds in a KeePass, did the data store joins, and federation and everything else and then disabled the PSA, never using it, except for upgrades or get out of jail.  This issue has come up, we've enabled the PSA again, without changing it and it still stays.  But we found the route issue was that the Portal URL was defined for the Relational Datastore, but not the Tile Cache Datastore in "Describedatastore.bat"  Once the Portal URL is present the issue seems to go away.  I actually wonder if it's a case of it can't authenticate with the Portal because it doens't know where it is.  Only a hunch though.

[FraserHand]

That is interesting - @MarcGraham2 did you have an owning portal url?

[MarcGraham2]

we didn't, but after unregistering and re-registering we did.