Federated server validation fails after migrating ArcGIS Enterprise 10.9.1 to a new Server

[SudhirPonnappan1]

Attempting to upgrade our current ArcGIS Enterprise 10.9.1 system to 11.1 and also migrate to a new 2022 Windows Server OS. This is a base install where all the enterprise GIS components (Portal, Server, and Datastore) are installed on the single machine. Following the steps outlined in the blog post https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/migrate-to-a-new-machine-...

Steps:

1. Configure a new VM with Windows Server 2022 OS.

2. Installed and configured ArcGIS 10.9.1 on the server. Used Chef automation and configured the system along with federation.

3. Installed all the patches and updated on both the Servers

4. Took a full backup of the existing enterprise GIS using the WebGISDR utility.

5. Imported the backup into the new Server. The import process ran successfully with out any errors. All portal content is available, the hosted feature layers are accessible and can be visualized.

** Only issue is that the ArcGIS relational datastore does not validate and the Federation is broken.
I have attempted this process in three different environments and encountered the same issue.

Attached are the screen shots with the validation errors. 

ArcGIS Server log message:
The connection property set was missing a required property or the property value was unrecognized. This release of the GeoDatabase is either invalid or out of date. DBMS table not found [GDB_Release]

Tried to unregister the datastore and reconfigure without any luck. Have an active support case with tech-support. Has anyone encounted a similar problem?

The error message logged doesn't make much sense as the GDB_Relase table does not exist in the database (New or the current functional enterprise GIS)

Any insights, troubleshooting tips or assistance will be greatly appreciated. 

Thanks!

[JakeSkinner]

@SudhirPonnappan1 are you able to upgrade the existing environment to 11.1?  I find with the latest release of the software, the webgisdr performs better.  I would try upgrading your existing environment to 11.1, create the webgisdr backup, install 11.1 in your new environment, and then try the import.

[SudhirPonnappan1]

@JakeSkinner- Thanks for the response.

Last night, I attempted this (well sort of), and the issue still persists.
Did not upgrade the existing 10.9.1 environment as this would have a significant downtime.

I got two new VM's setup.

1. On the first VM, installed 10.9.1 and imported the backup from the existing 10.9.1 environment. This broke the federation. However, I attempted to upgrade this broken 10.9.1 environment to 11.1, which was successful. Although the federation issue still persisted. Took a full back-up of this 11.1 environment.
2. Second VM, installed 11.1 and restored the back-up from the previous 11.1 upgrade. Again, the webgisdr import was successful. 

I was not very confident this would work. 

From what I can find on the error message logged in ArcGIS Server about the [GDB_Release] table not being found - mostly points to restoring the database/datastore with a different name (which does happen with webgisdr import). 

Not very keen on doing an in-place upgrade of the existing system unless there is no other option.

[JakeSkinner]

@SudhirPonnappan1 one other thing you can try if you have an additional server:

1. Install ArcGIS Data Store on the new server
2. Configure with ArcGIS Server (Data Store will be standby)
3. Make new ArcGIS Data Store primary
4. Remove old Data Store
5. Check validation

[SudhirPonnappan1]

@JakeSkinner 
Like you mentioned, I went ahead and setup a standby ArcGIS Datastore machine. Data got synced and I was able to swap the primary and secondary datastores. Federation still remains broken!
Do you believe the issue is still with the datastore or some configuration on ArcGIS Server? Everything validates properly on the Datastore including access to the hosted feature layers.