Portal License File Fails to Import after Upgrade

[Kathleen_Crombez]

We just upgraded our ArcGIS Enterprise deployment from 10.7.1 to 10.8.1 and are having issues importing the new license file generated in MyESRI.

From the Portal GUI I receive a generic message 'There was an error importing your licenses'.

From the Portal Admin API and from the command line, I receive a more detailed message 'Failed to configure the definition of the appBundles Essential Apps.'

In the Portal logs I see several SEVERE errors that read 'The server at https://machine.domain:7443/arcgis/sharing/rest/portals/configureAppBundle' returned and error. Invalid app Bundle. ["Invalid apps"]

Has anyone ran across a similar issue?

I have tried recreating the license file and i have already ran a repair on the portal installation.

Our deployment consists of the following:

Portal for ArcGIS	Microsoft Windows Server 2016 Standard
ArcGIS Server (Hosted)	Microsoft Windows Server 2016 Standard
ArcGIS Data Store	Microsoft Windows Server 2016 Standard
ArcGIS Server (with Image Server)	Microsoft Windows Server 2019 Standard

Thanks,

Kathleen

[Kathleen_Crombez]

This is an interesting thought.

Originally it was on a shared drive and I am pretty sure I was trying to access it via UNC path.

I did moved it to the local server and tried to import it from there... but we generally do not use the C drive.

Our IT sets up all our VMs with the OS installed on the C drive, then they create a D drive dedicated to all the software installations.

I do recall having issues installing the Web Adapter in the past if the installation files are extracted on the D drive instead of the C drive.

I will definitely keep this in mind as I attempt the installation again tomorrow.

Thank you for your comment and bringing this into my radar.

[Mandalorian]

@Kathleen_Crombez - I just upgraded my Portal from 10.7.1 to 10.8.1 and am having the exact same error appear when I attempt to Populate License via Portal Admin:

Interestingly, the Portal logs show a decryption error:

How did your reinstall go? Did it work out the second time?

[Kathleen_Crombez]

@Mandalorian,

I never did figure out what caused the issue the first time. But the second time I did the upgrade it was successful. (Granted it did not go as smooth as I would have liked, but it was a success.)

We uninstalled Cisco AMP (antivirus), disabled Sophos (antivirus and file scanning), and disabled Windows Defender prior to the second attempt to upgrade.

I kept a close eye on the Task Manager to make sure the upgrade was still running, because there are several instances where it seems to stall and isn't doing anything. This helped my sanity a little to see activity on the installer, so I knew it hadn't locked up.

At one point during the upgrade I had a message pop up that the windows installer was not responding. I was able to click to continue and it started again.

Once that long (4 hour) process finished I finally got to the point where you upload the license.json file and click to continue the upgrade of portal. after it finished the upgrade I received the following error:

Upgrade failed. java.lang.Exception: Failed to start the database server. The startup timed out. 

Interestingly, I had observed a bunch of PostgreSQL Server background processes starting up, but then they started shutting down before the error occurred. Additionally, there was only one OpenJDK Platform binary background process running when it failed.

I restarted the portal machine (waited a good 15 minutes or more to make sure the system was ready before proceeding) and attempted to upload the license.json file again to complete the upgrade. This time the PostgreSQL Server and OpenJDK Platform binary background processes seemed stable and did not shut down.

The upgrade then finished with a success message.

This was by far the most painful and stressful upgrade I have experienced of any software, ever. Portal alone took 6 hours to complete and the entire system upgrade took just under 12 hours. I never want to do this again. But unfortunately, we will need to upgrade to 10.9.1 to take advantage of the new parcel fabric features. ugh!

I wish you the best of luck!

[DavidColey]

Hi again Kathleen - just reviewing some posts and blogs in prep for the impending 9.0 upgrade and came across this again.  Yes, when you say:

"This was by far the most painful and stressful upgrade I have experienced of any software, ever. Portal alone took 6 hours to complete and the entire system upgrade took just under 12 hours. I never want to do this again. But unfortunately, we will need to upgrade to 10.9.1 to take advantage of the new parcel fabric features. ugh!"

I could not agree more.  Portal alone does take at least 6 hours.  For us, the entire upgrade time is beginning to approach 18 hours.  Portal, 2 hosting arcgis servers, 2 federated arcgis servers, relational data store, tile cache datastore plus the web adpators.  I would say the by far the most gut-wrenching part are the post-installs, where any kind of network failure at that point would hose the whole install.  Talk about panic filled hours. 

That's a long time to have a system offline.  ESRI needs to do something to speed  this up- please

[Kathleen_Crombez]

I am so sorry you are going through this right now. Your configuration has an additional hosting server and the more data in the datastore would certainly lead to a longer upgrade time.

Our datastore was virtually empty at the time I did our upgrade to 10.8.1 which helped keep our upgrade to just under 12 hours.

But we have been using the datastore more lately and this makes me very nervous about our impending doom when we upgrade to 10.9

Best of luck to you in your upgrade. I hope it doesn't give you too much heart burn.

[RoseZhou2]

I had the exact same issue when upgrade ArcGIS Enterprise from 10.7.1 to 10.8.1. It worked very well in our Dev. But got the same issue in our Staging environment. Checked with ESRI Tech Support, no workaround is offered. we are stuck here. So frustrated!!!!!

[Kathleen_Crombez]

Rose,

That is a shame.

Our only way to fix this issue was to roll back the servers and start over. We had to disable our anti-virus before attempting the re-install and it worked the second time.

Thanks,

Kathleen

[RoseZhou2]

Thanks Kathleen,

How did you roll back Portal only? I haven't started ArcGIS Server and Data Store upgrade yet. Did you follow the steps below to roll back Portal?

1. Uninstall Portal

2. Delete all existing ESRI directories

3. Reinstall 10.7.1

4. Restore from a 10.7.1 backup

It's so frustrated!!!

Regards

Rose

[Kathleen_Crombez]

We made checkpoints of all the servers prior to the upgrade, so we were able to restore the checkpoints we created. It you do not have checkpoints, I would recommend restoring the server from your last backup just prior to your upgrade.

[RoseZhou2]

Thanks A lot KathleenCrombez!