Upgrade to 10.5.1 and Resotring a 10.5 Site, Cautionary Tales

[danbecker]

Server was federated to Portal. Server was configured as the hosting server, Data store was also configured. Everything was working great at 10.5, but I wanted to upgrade, latest is greatest, right? This ended up being a 2AM nightmare. Rather then keeping these notes to myself, I thought I'd share.

Portal 10.5.1 Upgrade, Registered survey123 app

Portal (linux) update failed when attempting to create the initial site administrator account. Looking in the portal database logs, there was a problem when attempting to execute data.sql, specifically the INSERT statement dealing with survey123.

This is because I already had survey123 configured as a registered App so that we could use the survey123.arcgis.com website against our own portal. I understand this was in Beta with 10.5. As soon as I commented out that line in the data.sql file the initial site admin account could be created, no more errors.

CAUTION: If you have survey123 added as a registered application at 10.5, then attempt to 10.5.1 upgrade, you will likely experience the above error.

Portal 10.5.1 Upgrade, Webadaptor Re-install

After portal was upgraded to 10.5.1, I uninstalled/re-installed it's web adaptor. Same for Server, after it was upgraded to 10.5.1, I uninstalled/re-installed it's web adaptor. I logged into the Portal and tried to open an existing web map and would only get an IdentityManager popup, asking for credentials for the arcgisServer.local.domain machine? Server is federated with Portal, why would I bet directed to Server for a token?

After troubleshooting further, all previously published feature services were working correctly, no auth. popup box. It was the offline basemaps (actually webmaps) that I had configured. Why can't ArcGIS Enterprise users simply use ESRI basemaps offline like you can in ArcGIS Online? We have to complete this extra step, outlined last section of this page: Prepare maps for offline use—Collector for ArcGIS | ArcGIS 

I looked at my offline basemap(s) in Portal, details page, and the Service URL was arcgisServer.local.domain/portal. That's WAY wrong and would never be reachable outside our LAN. Not to mention it's the Server hostname and portal web adaptor name. Changing the URL to FQDNDNSAlias.com/portal worked right away, I was able to get to the rest endpoint of the basemap.

Devil is in the details. Look at this page: Configure ArcGIS Web Adaptor—Installation Guides (10.5) | ArcGIS Enterprise, Step #2. "If a DNS Alias will be used with the portal, the Web Adaptor should be configured over the DNS Alias instead".

So, after the portal web adaptor is installed, you have to access the web adaptor config. page on the machine it's installed on, but you have to use the same externally accessible DNS Allias, so https://myexternallyfacingFQDN.com/portal. Then you will reach the portal configure page where you enter the arcgisPortal.local.domain:7443 URL. After this step, the Service URL of any item you add to your portal via a URL will correctly display as https://myexternallyfacingFQDN.com/portal/item.....

Portal 10.5 Restore via RestAPI/Importsite

I cannot restore (or backup) using the portal webgisdr utility since my deployment is a mix of Linux/Win. So, I individually backup Portal using the exportSite REST API and backup Server using the serverbackup.py utility.

Initially I didnt' figure out the above portal web adaptor fiasco so I decided to uninstall portal 10.5.1 and go back to 10.5. Uninstall went fine, deleted the /arcgis/portal directory. Re-installed 10.5 and when I got to the initial admin. account setup, I entered the same initial admin username as before, portal_admin. I then went to arcgisPortal.local.domain:7443/arcgis/portaladmin/importSite. I specified the full path to the .portalsite file and waited, fingers crossed. After 10 minutes, Error. I looked in the portal/database logs and found an error related to INSERT "gw_" does not satisfy the unique key requirement, VALUES 'portal_admin'. To my novice SQL eye, this looks like the restore was attempting to INSERT username portal_admin from the backup, when I had already specified portal_admin during the intial admin. account config.

So, I uninstalled 10.5 and re-installed again, this time when I got to the initial admin. account config. I entered tmp_admin, then the /importSite operation succeeded!!

FYI, since Server/Portal were federated, I was a little unsure of the restore order, here's what I used and it worked great:

1. Uninstall Portal, Server, and both web adaptors
2. re-install Portal, configure a new/unique initial site admin account
3. re-install Portal web adaptor. Remember my DNS Alias point above, this is what got my into my 2AM nightmare.
4. re-install Server
5. re-install Server web adaptor
6. navigate to arcgisPortal.local.domain:7443/arcgis/portaladmin/importSite, specify the full path to the .portalsite file
7. Cross fingers. Our site is pretty small and it took about 30 minutes.
8. To my surprise, the importSite REST operation also restored all Server Services. I'm assuming this is because Server was federated.
9. I did not need to run the restore.py utility on the Server machine.

[PaulDavidson1]

Thanks, I wish more folks would share trials and tribulations. 

Have you considered the Esri Chef Cookbooks for deployment and upgrades?  They have proven very useful for us.  Haven't, we haven't jumped 10.4.1 to 10.5.1 yet. 

If you ever have the chance to move into a VM world for your servers, it could save you a lot of headaches.  You can snapshot before making changes so it's an easy recovery.  Easier to setup and rebuild servers, etc... and makes it easier to setup a Test environment so you hopefully work through these pains before upgrading Production. 

Thanks again!

[danbecker]

I just rely on the help/tutorial docs ESRI publishes; I find them to be very thorough. Honestly, the system components are so complex it would be difficult to document everything. For example, I couldn't find hardly any documentation on portal REST API exportSite and importSite. Lots of docs on webgisdr utility, but the docs fail to mention that Linux/Win mix of OS's isn't supported (that's for another thread).

A month ago we moved all 3 boxes to a single hyper-V core host. I supposed I could spin up another VM but resources are pretty taxed already. It would be expensive create a sandbox of the 3 VMs with MS and ESRI licensing being the biggest hurdle. I guess I could scrap WinServer1 for Ubuntu Linux3, it works just as well, if not better.

Ubuntu Linux1- Portal

Ubuntu Linux2 - PostgreSQL (SDE GDB backend) and Datastore

WinServer1 - Server and 2 web adaptors

WinServer2 - Domain Controller

Like you said, next time I'll take snapshots of each box before I take the plunge. But, I know what I did wrong and I'm sure when I upgrade next time, it will go smoothly.

[JonathanQuinn]

We're updating the documentation to be more explicit about the Web GIS DR tool and mixed OS's in a deployment so hopefully that's more clear.

In regards to the REST API for exportSite/importSite, the easiest way to get to it may be through the installed help:

https://<machine>.<domain>.com:7443/arcgis/portalhelp/apidocs/admin/index.html#/Export_Site/03240000... 

https://<machine>.<domain>.com:7443/arcgis/portalhelp/apidocs/admin/index.html#/Import_Site/03240000... 

[AlexZhuk]

I've been using ArcGIS Desktop since v 8.0. With every upgrade it was only getting better and better. Also I've been administering the AG Server since v. 9.3. And EVERY TIME, configuring it was a pain in the a... I, too, like the "latest is greatest" and am an early adopter of the ESRI software. But every time I fret when I have to upgrade the Server. It feels like buying a car that is not ready to drive off the lot, but instead has to be assembled by the buyer from separate pieces. Granted, there are many scenarios how and where the system can be installed and configured, and that's why it's not intuitive enough, but there are just got to be a simpler solution for the average JoeGIS. ESRI tech support staff, though, are knowledgeable and helpful people. But even they sometimes get confused.

[harleypowers_parks]

I have spent the entire week trying to figure out a safe path to upgrade a simple windows arcgis portal site from 10.5 to 10.5.1. I need to be able to upgrade our production site within a few hours, but still be able to rollback.  I have not been successful yet in a test environment.  ArcGIS Portal does complete the upgrade and the migration. But to see the content requires completing an reindex of the site. However, the "search" index becomes corrupted. So most of the content is missing. While I'm waiting for a way to fix the index... I'm going to try and take a smaller step.. and just attempt to  backup and recover the same server on 10.5, then see if I can upgrade to 10.5.1. 

now that I see you can uninstall and restore from the exported site, I am encouraged.

In reading thru your post, you installed a pristine version of 10.5.1, then ran the  import of the portal 10.5 exported site? 

you needed to use a different admin user name to complete the import? is that correct?

[harleypowers_parks]

So... we can't use webgisdr to upgrade

[KirkChesla]

Hi Harley,

Did you ever find a real solution for fixing the index in Portal?  Ours has also become out of sync when attempting to  upgrade 10.5.1 to 10.6.

[JonathanQuinn]

If you run a full reindex, does that fix the problem?

[Luiz_AmadeuCoutinho]

The upgrade process here got the same problems on Linux enviroment. We could´t upgrade yet.

Webgiddr didn´t work to backup our system. Exportsite only export datastore and Portal (not Server).

So, I´ll try your workflow . Thank you.

FYI, since Server/Portal were federated, I was a little unsure of the restore order, here's what I used and it worked great:

1. Uninstall Portal, Server, and both web adaptors
2. re-install Portal, configure a new/unique initial site admin account
3. re-install Portal web adaptor. Remember my DNS Alias point above, this is what got my into my 2AM nightmare.
4. re-install Server
5. re-install Server web adaptor
6. navigate to arcgisPortal.local.domain:7443/arcgis/portaladmin/importSite, specify the full path to the .portalsite file
7. Cross fingers. Our site is pretty small and it took about 30 minutes.
8. To my surprise, the importSite REST operation also restored all Server Services. I'm assuming this is because Server was federated.
9. I did not need to run the restore.py utility on the Server machine.