topic Re: Portal License File Fails to Import after Upgrade in ArcGIS Enterprise Portal Questions

Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Wed, 06 Jan 2021 15:24:29 GMT

We just upgraded our ArcGIS Enterprise deployment from 10.7.1 to 10.8.1 and are having issues importing the new license file generated in MyESRI.

From the Portal GUI I receive a generic message 'There was an error importing your licenses'.

From the Portal Admin API and from the command line, I receive a more detailed message 'Failed to configure the definition of the appBundles Essential Apps.'

In the Portal logs I see several SEVERE errors that read 'The server at https://machine.domain:7443/arcgis/sharing/rest/portals/configureAppBundle' returned and error. Invalid app Bundle. ["Invalid apps"]

Has anyone ran across a similar issue?

I have tried recreating the license file and i have already ran a repair on the portal installation.

Our deployment consists of the following:

Portal for ArcGIS	Microsoft Windows Server 2016 Standard
ArcGIS Server (Hosted)	Microsoft Windows Server 2016 Standard
ArcGIS Data Store	Microsoft Windows Server 2016 Standard
ArcGIS Server (with Image Server)	Microsoft Windows Server 2019 Standard

Thanks,

Kathleen

Re: Portal License File Fails to Import after Upgrade

amnoy_am — Mon, 04 Jan 2021 20:26:23 GMT

Kathleen,

First of all, verify you have a json file (license file) for ArcGIS Enterprise 10.8 and not the previous version.

Then make sure you clear the browser cache, including cookies. Load the portaladmin and try importing the license file again.

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Mon, 04 Jan 2021 20:30:18 GMT

Thank you.

Yes, the file is a .json and I have tried recreating it several time.

I have already tried to clear the browser cache and cleared the cookies.

I have also tried importing the file from the command line (using run as administrator) which does not use a browser and it fails with the error 'Failed to configure the definition of the appBundles Essential Apps.'

Re: Portal License File Fails to Import after Upgrade

DavidColey — Mon, 04 Jan 2021 21:48:00 GMT

When I look at your license.png attachment you've got something going on with 2 different Creator level groups. Why is that? Also, are you using Named User licenses or are you using Concurrent Use?

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Mon, 04 Jan 2021 22:23:03 GMT

We are using both concurrent and named users.

The different creator groups are due to different licensing types and expiration dates.

The 10 perpetual creators are our concurrent licenses and the 4 that expire are named users that we converted from AGOL.

Re: Portal License File Fails to Import after Upgrade

DavidColey — Tue, 05 Jan 2021 17:59:33 GMT

Ok so I'm guessing that having the different creator groups on concurrent and named user didn't cause an issue at 10.7.1? Because its my guess that portal doesn't know how to allocate the app bundles. It's like it needs a license file for the concurrent use and one for the named users. I'd go through support on this one.

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Tue, 05 Jan 2021 18:27:28 GMT

Yes. Thank you David.

I spent over 2 hours on the phone with ESRI Tech Support yesterday and they think that the anti-virus software that our IT department installed on all our machines recently interfered with the upgrade process.

Per our support ticket, ERSI concluded the following:

The Portal logs only referenced that is was unable to process the appbundles contained within the json license

No other logs related to functionality

We tried running the upgradeportal batch file located in the Portal tools folder

The utility completed successfully but said that the Portal has already been initialized

From this point Portal thinks that the upgrade completed successfully but has missed crucial parts of the upgrade.

We have reverted all our systems back to a pre-install state and plan to reattempt the upgrade with the anti-virus disabled in 2 weeks.

Which is a real shame as the entire enterprise upgrade took almost 12 hours to complete from beginning to end.

I will post an update after the second attempt to upgrade (with anti-virus disabled) is complete.

Thanks again for your help.

Re: Portal License File Fails to Import after Upgrade

DavidColey — Tue, 05 Jan 2021 22:26:47 GMT

Yes Kathleen, I wish I could have been of more help. Yes, it takes our system about 12 hours to finish as well, especially with a host site, a fed site, relational, tile cache and then the web adaptors. Its nerve wracking at best because as our system is always live.

Others seem to be having issues going from 10.7.1 to 10.8.1. I went from 10.7.1 to 10.8 and then 10.8 to 10.8.1 with no trouble, we have basically the same set up as you. That's why I don't think its related to any virus software, as long as the explicit firewall processes are allowed to run and the internal and dynamic TCP ports are open. But those setting didn't change from 10.7.1 to 10.8.1

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Wed, 06 Jan 2021 14:02:01 GMT

David,

Even though I was not able to resolve our issues, I appreciate your comments none the less.

I am curious to hear more about some of the other issues you have seen that people are having with the migration, and if they were able to over come them.

I was not able to get a single search engine hit on the error message we received. But perhaps issues others have experienced may be of help in getting to the bottom of it. (I will do some searches with less specific key words in the ESRI community to see what I come up with, but if you have seen some common issues, please do share.)

Also curious what anti-virus is running on your enterprise servers?

Our IT department installed Cisco AMP on all our servers on Christmas Day and our upgrade was attempted on New Years Day. So we haven't really had much time to determine if the anti-virus had any adverse effects on the enterprise deployment.

I sure hope I don't need to do a double upgrade like you did in order to get this done. We are trying to migrate our parcel fabric from Desktop to Pro so it is pretty important we get this upgrade done before we start mapping 2021 parcel changes.

Thanks again.

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Wed, 06 Jan 2021 15:23:26 GMT

I thought I might also mention (not sure if it is relevant or not) but we have portal collaboration set up with 2 ArcGIS Online accounts.

Re: Portal License File Fails to Import after Upgrade

DavidColey — Fri, 08 Jan 2021 18:39:48 GMT

Hi Kathleen, not sure - I ran across a few things in this topic We use McAfee.

Our EIT Security made an exceptions group for the exe processes that need to run without scan - But our EIT security group just made the exceptions group... it was not in place when I upgraded. I'm hoping that when I go to 10.9 that it makes things easier.

Also, make sure they have all the ports open that you need on all the machines.

https://enterprise.arcgis.com/en/system-requirements/latest/windows/arcgis-enterprise-overall-system-requirements.htm#ESRI_SECTION1_753566AB780B4EB9AD253CD654F1DB7Fhttps://enterprise.arcgis.com/en/portal/latest/install/windows/ports-used-by-portal-for-arcgis.htm

We went from 10.7.1 to 10.8 as part of my regular upgrades - I did not go to 10.8 just so I could then go to 10.8.1.

To me though, I don't know how virus software can interfere with licensing portal, unless the virus software is somehow blocking the json file type from being read. The appBundles are associated with the Creator and Viewer types, since you have different groups for both concurrent and named user - to me that is the issue. Something is not being recogonized. Have you opened your json file in notepad ++ and examined it? Are there change between the 10.71 and10.81 files?

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Mon, 11 Jan 2021 13:56:08 GMT

Thanks David.

I did look at the .json file in notepad++. I also used a .json viewer to compare the 10.7.1 and 10.8.1 license files side by side.

There were a few additional items in the new file to accommodate the new apps and functionality in the new release, however the format was exactly the same and seemed fine.

When I was on the phone with tech support we double checked all the ports and those were not the issue.

ESRI tech support does not think the anti-virus is preventing the licensing from being installed. They believe that the anti-virus corrupted the installation. I did receive the following error after the update process completed...

"The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines."

I restarted the portal machine and re-ran the update and received the same error. This is when i started looking deeper and turned off the Cisco AMP Endpoint service, and after running the update a third time I received a message that the update was successful.

However, Tech Support thinks there may have been internal issues with the install due to the software running while the install was being performed.

There was some weird behavior that I was not aware was an issue where I would log into the portaladmin directory and it would take me directly to the "populate license" screen. It would allow me to navigate the other directories if I manually entered the URL, but I was not able to navigate from the home directory, as the home directory would take me back to the "populate license" directory.

I assumed this was because the licensing failed. But ESRI said that should not be the case, and that I should be able to access the home directory even if it is not licensed.

Ultimately they determined that somehow the license cannot be read due to some internal issue and while the software thinks it installed correctly... some internal items are either missing or corrupt.

I have nothing more to go on at this time. But a new install will be attempted this weekend (hopefully with Cisco AMP completely uninstalled and not just disabled) so we can know for sure.

I am aware of the recommended anti-virus exclusions and we have them set on our other anti-virus software (Sophos) because it actually interfered with the services and would cause them to going into a permanent "stopping service" state that could only be resolved by rebooting the server. A reboot would fix the issue for about an hour and the services would get hit again and get stuck in the "stopping service" mode.

After speaking with Tech Support I started searching deeper and notice that ESRI has updated their documentation that now has conflicting information with our original set up.

This technical article was updated recently and does not match what was originally in it when we set up 10.7.1 https://support.esri.com/en/technical-article/000015732

I also found this guidance document which is not 100% consistent with the technical article above, nor do the directory paths for portal seem correct. https://trust.arcgis.com/en/customer-documents/ArcGIS_Enterprise_AV_Guidance.pdf

Thanks again,

Kathleen

Re: Portal License File Fails to Import after Upgrade

Anonymous User — Fri, 15 Jan 2021 02:26:27 GMT

@Kathleen_Crombez, I think i have seen this if the Portal JSON file is on a UNC path.

I opted to move it to the C: drive of the server first just to make the process as simple as possible.

Could this be the issue?

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Fri, 15 Jan 2021 15:39:38 GMT

This is an interesting thought.

Originally it was on a shared drive and I am pretty sure I was trying to access it via UNC path.

I did moved it to the local server and tried to import it from there... but we generally do not use the C drive.

Our IT sets up all our VMs with the OS installed on the C drive, then they create a D drive dedicated to all the software installations.

I do recall having issues installing the Web Adapter in the past if the installation files are extracted on the D drive instead of the C drive.

I will definitely keep this in mind as I attempt the installation again tomorrow.

Thank you for your comment and bringing this into my radar.

Re: Portal License File Fails to Import after Upgrade

Mandalorian — Wed, 27 Jan 2021 20:03:37 GMT

@Kathleen_Crombez - I just upgraded my Portal from 10.7.1 to 10.8.1 and am having the exact same error appear when I attempt to Populate License via Portal Admin:

Interestingly, the Portal logs show a decryption error:

How did your reinstall go? Did it work out the second time?

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Wed, 27 Jan 2021 22:40:39 GMT

@Mandalorian,

I never did figure out what caused the issue the first time. But the second time I did the upgrade it was successful. (Granted it did not go as smooth as I would have liked, but it was a success.)

We uninstalled Cisco AMP (antivirus), disabled Sophos (antivirus and file scanning), and disabled Windows Defender prior to the second attempt to upgrade.

I kept a close eye on the Task Manager to make sure the upgrade was still running, because there are several instances where it seems to stall and isn't doing anything. This helped my sanity a little to see activity on the installer, so I knew it hadn't locked up.

At one point during the upgrade I had a message pop up that the windows installer was not responding. I was able to click to continue and it started again.

Once that long (4 hour) process finished I finally got to the point where you upload the license.json file and click to continue the upgrade of portal. after it finished the upgrade I received the following error:

Upgrade failed. java.lang.Exception: Failed to start the database server. The startup timed out.

Interestingly, I had observed a bunch of PostgreSQL Server background processes starting up, but then they started shutting down before the error occurred. Additionally, there was only one OpenJDK Platform binary background process running when it failed.

I restarted the portal machine (waited a good 15 minutes or more to make sure the system was ready before proceeding) and attempted to upload the license.json file again to complete the upgrade. This time the PostgreSQL Server and OpenJDK Platform binary background processes seemed stable and did not shut down.

The upgrade then finished with a success message.

This was by far the most painful and stressful upgrade I have experienced of any software, ever. Portal alone took 6 hours to complete and the entire system upgrade took just under 12 hours. I never want to do this again. But unfortunately, we will need to upgrade to 10.9.1 to take advantage of the new parcel fabric features. ugh!

I wish you the best of luck!

Re: Portal License File Fails to Import after Upgrade

DavidColey — Wed, 31 Mar 2021 19:05:00 GMT

Hi again Kathleen - just reviewing some posts and blogs in prep for the impending 9.0 upgrade and came across this again. Yes, when you say:

"This was by far the most painful and stressful upgrade I have experienced of any software, ever. Portal alone took 6 hours to complete and the entire system upgrade took just under 12 hours. I never want to do this again. But unfortunately, we will need to upgrade to 10.9.1 to take advantage of the new parcel fabric features. ugh!"

I could not agree more. Portal alone does take at least 6 hours. For us, the entire upgrade time is beginning to approach 18 hours. Portal, 2 hosting arcgis servers, 2 federated arcgis servers, relational data store, tile cache datastore plus the web adpators. I would say the by far the most gut-wrenching part are the post-installs, where any kind of network failure at that point would hose the whole install. Talk about panic filled hours.

That's a long time to have a system offline. ESRI needs to do something to speed this up- please

Re: Portal License File Fails to Import after Upgrade

Kathleen_Crombez — Wed, 31 Mar 2021 19:45:14 GMT

I am so sorry you are going through this right now. Your configuration has an additional hosting server and the more data in the datastore would certainly lead to a longer upgrade time.

Our datastore was virtually empty at the time I did our upgrade to 10.8.1 which helped keep our upgrade to just under 12 hours.

But we have been using the datastore more lately and this makes me very nervous about our impending doom when we upgrade to 10.9

Best of luck to you in your upgrade. I hope it doesn't give you too much heart burn.

Re: Portal License File Fails to Import after Upgrade

DavidColey — Fri, 21 May 2021 17:25:02 GMT

Hi @Kathleen_Crombez - I am so sorry I never got a chance to reply to your question re our anti-virus. We use McAfee

Re: Portal License File Fails to Import after Upgrade

DavidColey — Fri, 21 May 2021 17:25:32 GMT

Were you ever able to get your issues resolved?