Upgrade high available portal from 10.9 to 11.1 in windows

[Thomas_Puthusserry]

Hi all

We have a working highly available Enterprise portal 10.9 in windows federated with ArcGIS Server. We are in the process of upgrading this to 11.1 and following steps outlined by @MichailMarinakis here: https://community.esri.com/t5/arcgis-enterprise-portal-questions/upgrade-high-available-portal-from-...

Installed Portal for ArcGIS in both primary and secondary VMs and then started the upgrade process in primary vm. Out of 8 steps, it failed at step 3 with an error: Failed to import data to the content directory correctly, Unable to unzip portal content.

One of the reasons is that 

In the current deployment, the portal contents and server directories including the config-store are stored in a storage account in Azure, not in the dedicated fileshare Virtual Machine. These contents are now copied over to the fileshare VM.

Then followed the steps below:

In the portaladmin, the folder path is updated from the current one:

storeaccountname.file.core.windows.net\\agsfileshare\\testenv\\portal\\content

to \\\\vm-fileshare\\agsfileshare\\testenv\\portal\\content using the REST api, and that updated. Portal restarted and working fine.

To update the directories used by ArcGIS Server , I followed the steps here

Only the cache and output folders I could update. Couldn't update the Jobs and config-store.

arcgis service account is a local account in the portal / server VMs and the windows services in these VMs are run by that user.

Does anyone have a similar issue or experience? Any help would be appreciated !

@JonathanQuinn any suggestion?

[Dan_Brumm]

Hey Tom, I run multiple HA Enterprise systems all at 10.9.1. We actually migrated all of our shared configs and contents directories to new locations this week. 

Is the share drive migration blocking you or is it the upgrade?

That error message (unable to import portal contents) you received when upgrading. I have hit many times.. For us we found that  Thumbs.db files in the Portal content folder cause it. Before we run an upgrade I have to go through and delete all of those files. 

[Thomas_Puthusserry]

Hi @Dan_Brumm Thanks for your response, and it gives me some courage that there are folks out there who have gone through this.

The issue here is upgrade. When I tried to upgrade yesterday, it went through until the above error appeared. I have checked the portal content directory and couldn't find any Thumbs.db files, so I thought it may be related to the location of the shared folders and thought to try that way.

When you upgraded the Pri and Standby portals, what was the steps followed? In my case I installed the 11.1 version in both VMs, then waited to see the installation completion message to appear in both VMs. Then I started with upgrade in Primary portal VM. While it was doing the update in Primary VM, I just clicked the completion message in the Standby VM, and loaded the localhost:7443/arcgis/home page. It said site created and login page appeared. I am not sure this action in Standby VM has caused any issues with the ongoing update process in the Primary VM. That is all I can think of I did differently.

Thomas

[Dan_Brumm]

Tom

Just a fun little nuance with those thumbs.db if you were not aware. Those are hidden from the search results by default. You have to go into File Explorer --> View tab --> options change folder and search options --> view tab --> scroll down under Advanced settings --> uncheck "hide protected operating system files.     Its also possible you don't have any and those files are just an issue with our system. 

Your upgrade steps sound like the same steps I take.  Did you look at the portal logs?  If you cant get to them through the PortalAdmin, you may have to dig into them on the C: drive

[Thomas_Puthusserry]

Thanks for that update @Dan_Brumm after going through steps you outlined, I could see around 42 instances of Thumbs.db files in the items folder. Have removed these files manually. 

I am keeping the contents in the Azure Storage account for now: storeaccountname.file.core.windows.net\\agsfileshare\\testenv\\portal\\content, as that is not seems to be the issue.

What is the process now? I am doing 11.1 updates on both portals (prim/standby) and will initiate the update process from primary portal VM as did last time. 

Thomas

[Thomas_Puthusserry]

Thanks for the above tip @Dan_Brumm the removal of the Thumbs.db files actually progressed with upgrade and managed to do it in the portal primary vm. The post-upgrade option completed as well in the same VM. 

Few issues:

1. After upgrade, the appgateway (we don't use web adaptors) is not picking the portal url. Portal loaded internally on 7443 port. Is this due to the internal self-signed SSLs changed during upgrade?

2. The standby portal did not load at all (internally on port 7443)

3. Not sure what has happened, but after a while the primary stops loading within the primary portal itself. 

On a positive note, upgrade happens in primary, but unable to understand what happens with standby upgrade.

Any help will be appreciated. can share log files if needed

Thomas

[ThomasColson]

@Thomas_Puthusserry  "I am keeping the contents in the Azure Storage account for now: storeaccountname.file.core.windows.net\\agsfileshare\\testenv\\portal\\content, as that is not seems to be the issue."

Would you care to share how you've gotten your Azure storage account to work as the Portal Content Directory? We have a case open with ESRI going into it's second month, and no one can figure out why we can't get it to work. 

[Thomas_Puthusserry]

@ThomasColson Our initial deployment of HA enterprise was using a customised script created originally from Azure Cloud Builder deployment artifacts. That option allows to select the Azure storage account to store the config store and portal contents: Check this video: https://www.youtube.com/watch?v=o5odrHBdzpA around 23:48. 

We found out that the performance was not good in such deployments and we migrated the folders to filesh.are VMs for production instances.

It is also to be noted that once Azure storage is taken as the location for the server config store, there is no option to edit that and change from server manager. That option was greyed out.

Thomas

[Dan_Brumm]

Hey Tom, 

1. Make sure you clear your browser cache. that can cause really odd behavior after upgrade. 
2.  The standby portal not upgrading correctly. I have run into this a few times. My Primary portal always seems to upgrade as expected. My quickest fix for this was to simply remove the standby portal from the site, reinstall the portal software from scratch and then join it back to the Primary.   I know it's not a great workflow, but it gets the job done. I have even tried removing the standby portal before I upgrade because it tends to go smoother. 
3. Check the portal logs and see what they look like.  Are there any errors that standout? 
4. A lot of times reboots help. 
5. Check Event Viewer and see if your Windows Firewall is blocking any traffic. 

[Thomas_Puthusserry]

Thanks @Dan_Brumm 

Is there any danger with unregistering a portal? is it straight forward to re-register it via the portaladmin API? Has there been any help /articles  from ESRI about the issues with Standby portal upgrade?