Issue with ArcGIS Server security-configuration.json after failed upgrade

[PhilippeVDV]

Hi folks

We upgraded from AGE 11.2 to AGE 11.5

Portal, ArcGIS Server, relational ArcGIS Datastore & tile cache ArcGIS Datastore on separate VMs (VMWare)

Portal content folder, arcgis server directories  & config-store on a fileshare.

Upgrade of Portal went fine

Upgrade of ArcGIS Server was not succesful

I found another post (Solved: Upgrade step 2 of 2 : Failed. Refresh auto-deploye... - Page 2 - Esri Community) that suggested to try another browser. However as ArcGIS Server was completely broken already, we needed to restore the snapshot of the VM. As the Portal upgade was succesful, I decided to only let restore the ArcGIS Server VM. After the restore we had an 11.5 Portal federated with a 11.2 ArcGIS Server. ArcGIS Server directories and the config-store on the fileshare were reverted to the 11.2 files.

Next, I did a second attempt to upgrade the 11.2 ArcGIS Server. But I saw an error in the ArcGIS Sever logs:

<Msg time="2025-09-15T16:36:34,941" type="SEVERE" code="6618" source="Admin" process="11988" thread="1" methodName="" machine="xyzxyzxyz" user="" elapsed="0.0" requestID="">Failed to return security configuration. Cannot invoke "com.esri.arcgis.discovery.admin.store.ConfigStore.securityConfigLastModified()" because "this.d" is null</Msg>

The timestamp of the error matches this step in the upgrade process:

When arrived at the Continue Server Upgade step, it failed again but other behavior now. After clicking Continue Server Upgrade, the process immediately terminated (1 second) and an authentication-prompt appeared. Seems that the federation / communication between Portal and ArcGIS Server was not OK anymore and after digging in the logs I found the error above. Most services are missing.

I try to understand what went wrong the 2nd time: error seems to point to the security-config.json, but that file was just present on the fileshare

No idea what the error  (Cannot invoke "com.esri.arcgis.discovery.admin.store.ConfigStore.securityConfigLastModified()" because "this.d" is null</Msg>) tries to say

Any idea what might be the issue? 11.5 Portal federation with 11.2 ArcGIS Server validated well (only complains about the version not matching beween Portal and ArcGIS Server but this makes sense during an upgrade).

What do you usually do when an ArcGIS Enterprise upgrade fails  in a later stage and the different components are on several VMs? As Portal upgrade went fine I decided to only restore the ArcGIS Server VM and to retry only the upgrade of ArcGIS Server (Portal upgrade took already 1.5 hours). Restore all VMs and start all over again with the Portal upgrade? 

Best regards

Phil

@JonathanQuinn @Scott_Tansley 

[Scott_Tansley]

Hi.  I’ve had a similar severity predicament recently.  I ended up uninstalling the software component.  Reinstalling at the current version.  Restoring the agssite file, testing/proving it and the. Upgrading on a fresh build.  A lot of work but it worked well.  

I haven’t seen your error before so can’t give specific advise on it.  Sorry.

Scott

[PhilippeVDV]

Hmm, that is a lot of work. I hope I'll  be able to find another solution (i.e. restore ArcGIS Server, complete the upgrade of ArcGIS Server and finally, upgrade the 2 Datastores)

[Scott_Tansley]

I hear you it was a lot of work.  Unfortunately, because I'm under contract and have to deliver within set timescales it's sometimes better to invest the time, clean up the server and approach it in that way rather than spend hours looking for a needle in a stack of needles.  I have to put tried and tested methods into play that will deliver, rather than trying to find the actual fix.  As an example, my most recent incident saw me unable to upgrade, repair or uninstall an Enterprise Portal.  I had to brute-force it's removal and implement the approach above to get the project over the line and not impact my forward programme of work.  Hard yards and drastic action but the client have now completed their UAT, and ultimately it was the most expeditious way of moving forward.  Once reimplemented as 11.3 (clean install and restore of content), the upgrade to 11.5 was a breeze, also the server and datastore upgraded in the way that I would have wanted them to.

I hope you solve this.  

[PhilippeVDV]

Hello Scott, I'm afraid you workaround (clean install of 11.2, import through webgisdr & upgrade to 11.5) will not work for us due to another bug: 

ArcGIS Enterprise 11.1-11.2: Restoring tile cache breaks scene layers

Customer has hosted scene layers and unfortunately we already faced the issues described in this post: restore of tile cache datastore fails in 11.2 + none of the workarounds provided by Esri worked for other customers. That bug would be fixed in 11.3 

[Scott_Tansley]

My recommendation was to clean install only the ArcGIS Server and restore from the agssite file.  There is no need to restore all components or to use webgisdr.

[PhilippeVDV]

Hi Scott

Just to be sure that we understand each other.

Do you mean the following procedure?

1) Uninstall the 11.2 ArcGIS Server (that is federated with a 11.5 Portal, after a successful Portal upgrade)

2) Empty all arcgis server directories (except arcgiscache) + config-store folders

3) Clean install of ArcGIS Server 11.2

4) Import of 11.2 agssite.file

5) Validation step

6) Upgrade of ArcGIS Server to 11.5

Best regards

Phil

[Scott_Tansley]

Yes.  The federation is contained in the agssite not the raw software.  I can't promise that this will work, but it puts you in a clean place and, hopefully, you still have VM backups at this point?  

[PhilippeVDV]

Hi @Scott_Tansley . I couldn't reply earlier. Your workflow described above worked for me. That is : after the successful upgrade of Portal to 11.5 and the failed upgrade of ArcGIS Server to 11.5 the ArcGIS Server was completely broken; after following the procedure you suggested (uninstall ArcGIS Server 11.5, clean install of ArcGIS Server 11.2 and import of the 11.2 agssite file, I had an 11.5 Portal federated with an 11.2 ArcGIS Server. That is the good news. But when upgrading the 11.2 ArcGIS Server directly to 11.5, we see again the same error (Failed to return security configuration. Cannot invoke "com.esri.arcgis.discovery.admin.store.ConfigStore.securityConfigLastModified()" because "this.d" is null</Msg>). The only way to workaround it is to upgrade to ArcGIS Server 11.3. Then no error at all.  Unfortunately, when upgrading beyond 11.3, then we still have the other error (Upgrade step 2 of 2 : Failed. Refresh auto-deployed services during upgrade). Esri inc support has no clue about both issues yet . (The step with upgrading to 11.3 was our own idea). So now we have an 11.5 Portal federated with an 11.3 ArcGIS Server with a registered 11.2 relational Datastore and an 11.2 tile cache Datastore. But as we can get rid of the first error, fine for me to upgrade ArcGIS Server first to 11.3 before proceeding to 11.5. Hopefully Esri inc finds a solution for the 2nd issue

[A_Wyn_Jones]

Is ArcGIS Server fully patched at 11.2? 

It seems like a permissions error related to your file share, for a sanity check - i would run this tool against the ArcGIS Server, using the same service account:

https://enterprise.arcgis.com/en/server/11.2/get-started/windows/what-s-included-with-arcgis-server....

I would also check you don't have any active replication on your fileshare which may be affecting the file system....namely DFS: https://enterprise.arcgis.com/en/server/latest/deploy/windows/choosing-a-nas-device.htm#:~:text=and%...