error when generating tokens for ArcGIS Server 10.8.1 (HA)

[SaurabhUpadhyaya]

In our production environment, we have two ArcGIS Web Adaptors behind a load balancer to serve a high-availability site. We have two physically separate ArcGIS Server sites (Server1 and Server2) mirrored and attached to a Network Load Balancer (NLB) with the URL https://sample.gisservices.com. Each site serves the same secured services and uses different shared and public keys for token generation because we have not configured second server using existing site. It is working because no security implemented and request is handled by load balancer.

We are now planning to secure these services using tokens.

We use a generateToken mechanism via https://sample.gisservices.com/arcgis/tokens/generateToken and try to access the secured services using the ArcGIS JavaScript API with URLs like https://sample.gisservices.com/arcgis/rest/services/sample/MapServer with the token. However, we encounter an issue where sometimes the map layer loads successfully and sometimes we receive an "invalid token" error, causing the layer not to load.

ArcGIS Server Versions : 10.8 on both machine.

Kindly suggest a solution to resolve this issue on priority.

[viopeji]

You web adaptor allows you to point it at any server as long as you keep the suffix the same when you install it (web adaptors can't be upgraded - you have to remove it and install the new version). If you want to minimize downtime and/or prototype the new server, you can go through the trouble of creating a new VM and upgrading that instead of your existing AGS machine but it isn't really necessary. Upgrading AGS generally is pretty reliable. The two things that always ruin my upgrade are either Portal or DataStore. If you don't have to deal with those, it should be pretty easy and uneventful.

[vipulsoni]

Hi,

Check this out , this might help your case - Shared Key (from ArcGIS Server online guides) https://enterprise.arcgis.com/en/server/latest/install/windows/single-machine-high-availability-acti...

Using token-based security

If using token-based authentication, also referred to as server-tier authentication, it is important that all sites in this configuration use exactly the same shared token key. Otherwise, tokens generated for one machine will not be valid when used against the other machine. To duplicate the shared token keys across multiple sites, you can edit token settings in Manager.

Edit token settings in Manager

[SaurabhUpadhyaya]

Hi @vipulsoni @viopeji 

Thanks for your response.

Here i want to clarify few things:

1. We have 2 own premised ArcGIS Server 10.8, that web adapter is configured on NLB.

2. Both ArcGIS Servers are independent and configured independently means both ArcGIS Servers shared key, public keys are different ( This is the issue generated token tokens are not matching)

I connected with ESRI team, they are asking to try below solution but they are not confident it will work or not.

ESRI Support Solution 1: Changing the publickey to match on both Server sites.

• Stop ArcGIS Server services on all machines in the high availability (HA) Active site.
• Rename ..\arcgisserver\config-store\.site\rsa.ser to ..\arcgisserver\config-store\.site\rsa.ser.old on all secondary machines.
• Rename ..\arcgisserver\config-store\.site\rsa.ser.rlock to ..\arcgisserver\config-store\.site\rsa.ser.rlock.old on all secondary machines.
• Copy ..\arcgisserver\config-store\.site\rsa.ser from the master site to the secondary machines.
• Copy ..\arcgisserver\config-store\.site\rsa.ser.rlock from the master site to the secondary machines.
• Restart ArcGIS Server and confirm all machines now have the same publicKey.

ESRI Support Solution 2:  Changing the Load Balancer algorithm.

• We can work with the IT department to change the algorithm on the load balancer to use a "sticky" algorithm instead of a "round robin". The sticky algorithm stays with the same machine for the entirety of a session.

Please suggest me what should need to do? 

Do I need to install a fresh installation of arcgis server, join existing site (Server1), use shared config store(Copy Serve1 config store on share path and use in new server)?  If i will do this how load balancer will work?  pls provide me approach what should i need to do???

[vipulsoni]

hi @SaurabhUpadhyaya ,

I would suggest, take a VM Snapshot backup of the server image of the secondary server and test the Solution 1 from ESRI Support.

[JonathanQuinn]

Esri Support is correct; the problem is that each site has a unique key which is used to decrypt tokens. A token generated and encrypted on site a can't be decrypted by site b. Sticky sessions can work at the LB, although you won't get the benefit of round-robin requests to each machine for redundancy/horizontal scaling. A multi-machine server site will also work, but you need a shared location for the config-store and directories. This introduces a single point of failure but eliminates some hurdles regarding publishig and synchronizing services between the sites, because both machines are part of the same site. We have documentation on active/active siloed environments, and describe what to do for tokens here which you can follow if you still want an active/active configuration. Note that this can't be federated to Portal for ArcGIS.

[SaurabhUpadhyaya]

Hi @JonathanQuinn @vipulsoni 

We have followed all the steps outlined for the active/active configuration(https://enterprise.arcgis.com/en/server/latest/install/windows/single-machine-high-availability-acti...). of ArcGIS Server 10.8, and everything has been working well until we introduced security measures. Currently, nothing is on the shared drive (ArcGIS Server config store) as we haven't implemented the token concept yet.

Since this is a production environment with many users, we want to avoid any risks.

@JonathanQuinn, could you please help us by sharing the exact steps or solutions we need to implement? We prefer not to experiment in the production environment.

Here are the specific areas where we need guidance:

1. How to match the shared key.
2. How to match the public key, as it is also not the same.
3. Copying the primary site (Server1) config store and pasting it into the secondary site config store.
4. Sticky sessions.
5. Or any better approach?

Thank you for your assistance.

[SaurabhUpadhyaya]

Hi all,

can anyone share the solution or workaround to resolve this issue??