Portal 10.9.1 to 11.2 Upgrade Postgres Failure

[RandyBonds_Jr_]

We have repeatedly tried to upgrade Portal 10.9.1 to 11.2 with no luck. We've opened a support ticket with no solution as of yet. 

The process:

1. We take a backup
2. We run the 11.2 setup, when prompted for the Windows Service Account, it is grayed out with what is the portal is currently running at (our_domain\!arcgis). We enter the password, etc, and everything is great.
3. When on the next upgrade step (the web based part), the PostgreSQL upgrade fails, we see two items in the logs that seem to point to the issue:
   1. Copying old pg_xact to new server *failure* 
   2. FATAL: role "!arcgis" does not exist

Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for user-defined encoding conversions              ok
Checking for user-defined postfix operators                 ok
Checking for incompatible polymorphic functions             ok
Creating dump of global objects                             ok
Creating dump of database schemas
  gwdb
  postgres
  template1
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok
Checking for new cluster tablespace directories             ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           
*failure*

Consult the last few lines of "pg_upgrade_utility.log" for
the probable cause of the failure.
Failure, exiting

It appears that the copying failure is causing the issue of the incorrect user being used. Looks like it is defaulting to using the OS user when it later tries to start the DB. 

Anyone seen this?

[TimoT]

Hi @RandyBonds_Jr_ 

I'm guessing you are currently attempting to upgrade portal while logged in to the machine as !arcgis user.

My best guess is that the postgres upgrade is issuing pg_upgrade commands in a manner that doesn't explicitly specify a user, meaning it will simply try to use the current logged in user. Upon initial install of Portal, the postgres installation would likely have done something similar and created a username based on the current logged in user in its database.

I can think of a few ways to get around your current issue, but only the first in the below list may be considered supported.

1. Login to the Portal machine with the initial user that installed Portal, then run the upgrade. Might help if this user is part of the local Administrators group on the machine.
2. Create a new user in the postgres database to match what's missing in the error logs.
3. Set Trust Authentication in pg_hba.conf file. Doing this can be used to effectively bypass database authentication regardless of the username supplied.

I would say option number 1 is your best bet without Support guidance. Support may not endorse the other 2 options.

Timo

[RandyBonds_Jr_]

We are doing #1, this is the only user on that box (essentially) 

I'm not sure about 2 and 3, but it's really looking like the Transaction log upgrade is failing 

Copying old pg_xact to new server 

[DanielBrumm1]

I had a thought about this. It's a total guess.  Did anything change with your default Portal Database admin account? Could the password have changed?  Maybe try going into the Portal Administrator and resetting the database account password and retry the upgrade. 

PortalAdmin --> home --> System--> Database--> Update Admin Account

[RandyBonds_Jr_]

We have never changed the Portal DB Admin, but this may be worth a try

[RandyBonds_Jr_]

Our system admin was able to find another clue from the upgrade log:

pg_upgrade_utility.log from d:\arcgisportal\temp

command: xcopy /e /y /q /r "D:/arcgisportal/db1709936077740/pg_xact" "D:/arcgisportal/db/pg_xact\" >> "pg_upgrade_utility.log" 2>&1
Invalid drive specification
0 File(s) copied

This may be the root of the problem.

Still don't know how to fix it... 

[MikaelJ]

We ran into the same issue with 10.9.1 to 11.1.

Have you managed to achieve any progress on this topic (Invalid drive specification)?

[RandyBonds_Jr_]

We haven't yet. We have escalated with ESRI, hoping to hear back soon.

[QiranLiang]

We just encountered the same error from 10.9.1 to 11.2. Funny thing is the last attempt we don’t have this issue but we have a different issue about the indexer