Hosting Server (portal) Post Upgrade (10.8.1 to 10.9.1) extremly slow - How to speed up?

[LorenzMeyer1]

Yesterday we started a dry run to upgrade our Portal for ArcGIS with version 10.8.1 to 10.9.1.

The "post upgrade" (upgradeserver.bat) from the hosting server took almost 10 hours to finish. This is unusable!!!!

The portal consists of two windows servers 2016, one for the web adaptor and the other one for the portal itself (portal, hosting server, data store (relational & tilecache)). The portal itself contains over 1000 hosted feature layer and other items (~4300 items).

The portal server has 8 CPUs and 64GB of RAM. 

Why does the "upgradeserver" take this long time?? I mean, it does nothing on the server. No CPU peaks, no memory peaks, nothing. 

In the Esri documentation it's only written that it can take some time, depending on the amount of content. But almost 10 hours? Really?

Anybody else facing this slow upgrade? 

Anybody an idea on how to speed up this post upgrade process?

[Scott_Tansley]

Maybe a random question, but are you using AWS for cloud and GP2 for disk?

[LorenzMeyer1]

Hi Scott

No, but it's a private Cloud based on Openstack.

[Scott_Tansley]

I knew it was a slim chance.  AWS GP2 disk has limited IOPS and a burst 'feature'.  the upgrade process uses more than the IOPS available and it literally runs out of steam.  It's necessary to upgrade to IO1 disk to get the performance for the upgrade and then (possibly) downgrade.  Just thought i'd ask on the off chance.  You may want to check performance of disk though.  I've done 3 10.8.1 to 10.9.1 upgrades with significantly less times than what you've experienced.

[jcarlson]

In my experience over the past 4 upgrades of our Portal, that post upgrade command was one of the longest wait times for us as well, and has gotten longer the more layers we've added.

I've heard from other users that this does can take a good number of hours, so 10 doesn't sound terribly far-fetched. I was similarly vexed by the documentation seeming to understate the duration of this step. I even submitted an Idea for them to tweak that text to say "possibly several hours".

I would suppose the best way to speed this up is to keep your Portal nice and lean, removing unused and deprecated layers wherever possible.

[LorenzMeyer1]

This is really frustrating...

"config-store" and "server directories" all locally on a drive 😧 (SSD Ultra High I/O). Disabled all monitoring tools (including Windows defender), installed a tool to monitor file and folder activity (https://www.howtogeek.com/205144/how-to-monitor-file-and-folder-changes-in-windows/) and then started again the upgradeserver.bat.

Sometimes the config-store or a server directory (and files) is changed and then again waiting minutes and hours and nothing happens on the file system. So wtf does this tool??? And not even a single log file for this tool to inspect...   😞

[DavidColey]

I commented THIS WEEK on this here:

https://community.esri.com/t5/arcgis-enterprise-questions/enterprise-portal-10-9-1-post-upgrade-time...

All of our other portal post-upgrade step from 10.6 to 10.6.1; from 10.6.1 to 10.7,; from 10.7 to 10.7.1 and so on each took about 4 hours.  Fairly reasonable with a few hundred hosted feature layers and some 18k total items. 

This time, I gambled and killed the post-upgrade after 8 hours.  I believe it was hung up but I really don't know.

I DO KNOW that for all future Enterprise upgrades, I will not try to complete it on a single Sunday, but give myself an entire weekend.  There is nothing more nerve-wracking than waiting for that step to complete, knowing that I still have to apply 2 hosting server upgrade and post upgrade, web adaptor, 2 data stores, and 2 federated servers.

This was the first time I delayed updating the federated site.

[EricAnderson14]

I am in a similar boat. I am in the "Post Portal" Step as well and just now 10 hours in and no-end in sight. Sigh. 

Knowing this could take hours/days would have been useful information so I could tell the users what to expect. Sigh. 

Keeping my fingers crossed this process will complete sometime soon. 

Eric

[EricAnderson14]

After working with two different Support Analyst at Esri, the second one suggested that we try copying the URL from the Web window showing the Dialog box, to another instance of a browser window. This instance showed the process was complete, while the initial window showed the post process was still running. 

Of course, I wonder how long the process had truly been completed. Maybe it had finished 5 minutes earlier maybe it had finished 8 hours earlier. For this install, I may never know. 

Take care all! 

[LorenzMeyer1]

The complete post-upgrade process is a black box. You have no clue what's upgraded and you have no idea on how long it will take to finish. The web interface does not react. That's why I use the batch file "upgradeserver.bat" for this purpose and I added some "echo" and wrote out the start and end time. So I have at least a duration on how long it took for the process.