Issue with ArcGIS Server security-configuration.json after failed upgrade

PhilippeVDV

Hi folks

We upgraded from AGE 11.2 to AGE 11.5

Portal, ArcGIS Server, relational ArcGIS Datastore & tile cache ArcGIS Datastore on separate VMs (VMWare)

Portal content folder, arcgis server directories & config-store on a fileshare.

Upgrade of Portal went fine

Upgrade of ArcGIS Server was not succesful

I found another post (Solved: Upgrade step 2 of 2 : Failed. Refresh auto-deploye... - Page 2 - Esri Community) that suggested to try another browser. However as ArcGIS Server was completely broken already, we needed to restore the snapshot of the VM. As the Portal upgade was succesful, I decided to only let restore the ArcGIS Server VM. After the restore we had an 11.5 Portal federated with a 11.2 ArcGIS Server. ArcGIS Server directories and the config-store on the fileshare were reverted to the 11.2 files.

Next, I did a second attempt to upgrade the 11.2 ArcGIS Server. But I saw an error in the ArcGIS Sever logs:

<Msg time="2025-09-15T16:36:34,941" type="SEVERE" code="6618" source="Admin" process="11988" thread="1" methodName="" machine="xyzxyzxyz" user="" elapsed="0.0" requestID="">Failed to return security configuration. Cannot invoke "com.esri.arcgis.discovery.admin.store.ConfigStore.securityConfigLastModified()" because "this.d" is null</Msg>

The timestamp of the error matches this step in the upgrade process:

When arrived at the Continue Server Upgade step, it failed again but other behavior now. After clicking Continue Server Upgrade, the process immediately terminated (1 second) and an authentication-prompt appeared. Seems that the federation / communication between Portal and ArcGIS Server was not OK anymore and after digging in the logs I found the error above. Most services are missing.

I try to understand what went wrong the 2nd time: error seems to point to the security-config.json, but that file was just present on the fileshare

No idea what the error (Cannot invoke "com.esri.arcgis.discovery.admin.store.ConfigStore.securityConfigLastModified()" because "this.d" is null</Msg>) tries to say

Any idea what might be the issue? 11.5 Portal federation with 11.2 ArcGIS Server validated well (only complains about the version not matching beween Portal and ArcGIS Server but this makes sense during an upgrade).

What do you usually do when an ArcGIS Enterprise upgrade fails in a later stage and the different components are on several VMs? As Portal upgrade went fine I decided to only restore the ArcGIS Server VM and to retry only the upgrade of ArcGIS Server (Portal upgrade took already 1.5 hours). Restore all VMs and start all over again with the Portal upgrade?

Best regards

Phil

@JonathanQuinn @Scott_Tansley

PhilippeVDV

Thanks for your reply

Yes, 11.2 was fully patched

I'll check whether the permissions on the folders changed after restoring the 11.2 arcgis server directories + config-store) + will check about the active replication

What is your opinion on restoring snapshots when the upgrade fails in a later stage of the upgrade (here: upgrade of ArcGIS Server failed). Restore all snapshots of all machines and beginning all over again with the upgrade of Portal? Or is it OK to keep our upgraded Portal and only restore the snapshot of the ArcGIS Server?

DavidColey

The only time I have had a failed ArcGIS Server upgrade was here:

https://community.esri.com/t5/arcgis-enterprise-questions/arcserver-hosting-site-11-2-won-t-create-t...

When we were upgrading from 11.1 to 11.2. For whatever reason the ArcGIS Server.exe windows service could not be created.

We rolled back the Portal server and the Hosting Site ArcGIS Server servers from their respective snapshots. I then cleaned/upgraded/deleted/re-created any corrupted or otherwise mal-behaving items and hosted feature layers.

I recommend you do the same if you have not already. Of I made sure Server requirements were in place (i.e dot net) and then most importantly, I convinced my IT security team to allow me to disable active virus scanning during the re-upgrade.

This alone reduced what was about a 3-4 upgrade and post-upgrade time down to 30 minutes per machine for my 2-machine cluster Host site. Disabling AV during also meant that I no longer had any issue with the domain arcgiserver account needed to run the windows services.

PhilippeVDV

Hi David

Yes, I always ask IT to disable AV during upgrades. We restored all snapshots (not only ArcGIS Server machine). But after a new upgrade I see now again the first error: ArcGIS Server upgrade step 2 of 2 : Failed. Refresh auto-deployed services during upgrade’ . Recently, other people faced the same with 11.5: https://community.esri.com/t5/arcgis-enterprise-questions/upgrade-step-2-of-2-failed-refresh-auto-de... I'll discuss that issue in the respective post