Portal License File Fails to Import after Upgrade

9769
29
01-04-2021 10:20 AM
Labels (1)
Kathleen_Crombez
Occasional Contributor III

We just upgraded our ArcGIS Enterprise deployment from 10.7.1 to 10.8.1 and are having issues importing the new license file generated in MyESRI.

From the Portal GUI I receive a generic message 'There was an error importing your licenses'.

From the Portal Admin API and from the command line, I receive a more detailed message 'Failed to configure the definition of the appBundles Essential Apps.'

In the Portal logs I see several SEVERE errors that read 'The server at https://machine.domain:7443/arcgis/sharing/rest/portals/configureAppBundle' returned and error. Invalid app Bundle. ["Invalid apps"]

Has anyone ran across a similar issue?

I have tried recreating the license file and i have already ran a repair on the portal installation.

 

Our deployment consists of the following:

Portal for ArcGISMicrosoft Windows Server 2016 Standard
ArcGIS Server (Hosted)Microsoft Windows Server 2016 Standard
ArcGIS Data StoreMicrosoft Windows Server 2016 Standard
ArcGIS Server (with Image Server)Microsoft Windows Server 2019 Standard

 

Thanks,

Kathleen

29 Replies
DavidColey
Frequent Contributor

Hi Kathleen, not sure - I ran across a few things in this topic We use McAfee. 

Our EIT Security made an exceptions group for the exe processes that need to run without scan - But our EIT security group just made the exceptions group... it was not in place when I upgraded.  I'm hoping that when I go to 10.9 that it makes things easier.

Also, make sure they have all the ports open that you need on all the machines.

We went from 10.7.1 to 10.8 as part of my regular upgrades - I did not go to 10.8 just so I could then go to 10.8.1.  

To me though, I don't know how virus software can interfere with licensing portal, unless the virus software is somehow blocking the json file type from being read.  The appBundles are associated with the Creator and Viewer types, since you have different groups for both concurrent and named user - to me that is the issue.  Something is not being recogonized.  Have you opened your json file in notepad ++ and examined it?  Are there change between the 10.71 and10.81 files?

 

Kathleen_Crombez
Occasional Contributor III

Thanks David.

 

I did look at the .json file in notepad++. I also used a .json viewer to compare the 10.7.1 and 10.8.1 license files side by side.

There were a few additional items in the new file to accommodate the new apps and functionality in the new release, however the format was exactly the same and seemed fine.

When I was on the phone with tech support we double checked all the ports and those were not the issue.

 

ESRI tech support does not think the anti-virus is preventing the licensing from being installed. They believe that the anti-virus corrupted the installation. I did receive the following error after the update process completed...

 

"The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines."

 

I restarted the portal machine and re-ran the update and received the same error. This is when i started looking deeper and turned off the Cisco AMP Endpoint service, and after running the update a third time I received a message that the update was successful.

However, Tech Support thinks there may have been internal issues with the install due to the software running while the install was being performed.

 There was some weird behavior that I was not aware was an issue where I would log into the portaladmin directory and it would take me directly to the "populate license" screen. It would allow me to navigate the other directories if I manually entered the URL, but I was not able to navigate from the home directory, as the home directory would take me back to the "populate license" directory.

I assumed this was because the licensing failed. But ESRI said that should not be the case, and that I should be able to access the home directory even if it is not licensed.

Ultimately they determined that somehow the license cannot be read due to some internal issue and while the software thinks it installed correctly... some internal items are either missing or corrupt.

I have nothing more to go on at this time. But a new install will be attempted this weekend (hopefully with Cisco AMP completely uninstalled and not just disabled) so we can know for sure.

 

I am aware of the recommended anti-virus exclusions and we have them set on our other anti-virus software (Sophos) because it actually interfered with the services and would cause them to going into a permanent "stopping service" state that could only be resolved by rebooting the server. A reboot would fix the issue for about an hour and the services would get hit again and get stuck in the "stopping service" mode.

 

After speaking with Tech Support I started searching deeper and notice that ESRI has updated their documentation that now has conflicting information with our original set up.

 

This technical article was updated recently and does not match what was originally in it when we set up 10.7.1 https://support.esri.com/en/technical-article/000015732

I also found this guidance document which is not 100% consistent with the technical article above, nor do the directory paths for portal seem correct. https://trust.arcgis.com/en/customer-documents/ArcGIS_Enterprise_AV_Guidance.pdf

 

Thanks again,

Kathleen

0 Kudos
DavidColey
Frequent Contributor

Hi @Kathleen_Crombez - I am so sorry I never got a chance to reply to your question re our anti-virus.  We use McAfee

0 Kudos
DavidColey
Frequent Contributor

Were you ever able to get your issues resolved?

0 Kudos
Kathleen_Crombez
Occasional Contributor III

Yes. We successfully got our ArcGIS Enterprise upgraded from version 10.7.1 to 10.8.1 by uninstalling Cisco AMP and disabling Sophos anti-virus.

However, we recently had a windows update that suddenly caused the Sophos anti-virus on the ArcGIS Server machine to start killing off ArcSOC.exe processes.

We had all the exclusions set per the ESRI documentation, but it was still destroying the processes and leaving them as abandoned or orphaned processes causing the server to start using excessive amounts of memory.

After creating a case with ESRI support we decided (much to our IT departments dismay) to uninstall the Sophos anti-virus and all the issues went away.

I think it is pretty clear that ArcGIS Enterprise software is highly susceptible to issues caused by certain anti-virus softwares. Hopefully ESRI will put out recommendations and guidance for an anti-virus solution that is less destructive to the ArcGIS Enterprise functionality in the near future.

0 Kudos
DavidColey
Frequent Contributor

Hmm.  I do see that I actually did mention we use McAffee in an earlier reply.  So you went from Cisco to Sophos and even after you set up the exe and process exclusions Sophos still killed SOC processes?  Wow.  In our case, McAffee will sometimes orphan a SOC.  Or a SOC could be orphaned from a Nessus port scan process (that's a weekly thing our IT does).

The interesting thing in our case is that I can go in and stop the ArcServer exe and the SOC processes will sometimes still show as running.  But for McAffee this has thus far been a fairly rare (2-3 times a year?) occurrence.

Yes, your last paragraph here would certainly be helpful -

I am moving to 10.9 on Sunday.  I'll keep you posted.  If you hadn't seen my post on Map Viewer and Field maps on whether or not they should be un-installed first:

https://community.esri.com/t5/arcgis-enterprise-questions/fieldmaps-for-arcgis-at-enterprise-10-9/td...

I also just posted a question here

https://community.esri.com/t5/arcgis-enterprise-portal/portal-license-file-and-field-maps/td-p/10605...

where I am not seeing Field Maps in the 10.9 license file json

0 Kudos
DavidColey
Frequent Contributor

Hi @Kathleen_Crombez  - wanted to let you know that while the update did take me 14 hours, it went well. Portal and it's web adaptor took 5 - 6 hours, and then our 2 hosting servers and web adaptor took about 3 hours each.  I would say the time was about 2 hours for the software upgrade, then an hour each for the post upgrade.  Both our relational and tile cache stores updated in under an hour (thankfully I could do those in parallel).  Then our federated site that hosts an image server role was fast, about 35 minutes per machine.  

Once that was complete I reinstalled the Web Styles and Map Viewer for 10.9  Field Maps must be un-installed, and then it's new version installs with the 10.9 portal exe.

Everything is functioning so far, the Patch Notification tool on portal has a hiccup.  I did have to re-share some Living Atlas content, and I definently had to run the 'update' process from Portal Admin system properties for to re-establish the the web context :

https://myportal:7443/arcgis/portaladmin/system/properties/update

Otherwise, my admin and token generation DNS urls 'bounced' back to the machine name prefixes

0 Kudos
Kathleen_Crombez
Occasional Contributor III

Wow... I am not looking forward to this upgrade.

I am glad everything went (mostly) smooth for you!

Thank you for the additional hints on your workflow.

Hopefully ESRI will have any bugs you encountered worked out in a 10.9.1 release.

0 Kudos
DavidColey
Frequent Contributor

yes, i forgot to run the configuredatastore step on the relational datastore.  Maybe that will calm down the token error logging I'm getting

0 Kudos
by Anonymous User
Not applicable

@Kathleen_Crombez, I think i have seen this if the Portal JSON file is on a UNC path.

I opted to move it to the C: drive of the server first just to make the process as simple as possible.

Could this be the issue?