I experienced the same problem but with a different error message and solution.
A 10.4.1 GDB that I inherited passed the 'pre-req' checks but produced the following error message on the dialog window after trying to upgrade the GDB. The same message was in the sde_setup.log.
ERROR in creating GDB_Locks table.
Error: -25
DBMS error code: 0
Upon investigation, I discovered that the <My GDB name>.sde.GDB_Locks table was missing. Why? No idea, it's still a mystery.
The solution was to go into SSMS (SQL Server Management Studio) and add the 'db_owner' role to the 'sde' user. Then I ran the upgrade and revoked the 'db_owner' role from the 'sde' user.
This ESRI doc helped point me in the right direction, Privileges for geodatabases in SQL Server—ArcGIS Pro | Documentation
This solution may help another GIS DBA experience a similar problem.
Ciao!