"Reindex" only zeroes out the Portal Index (11.1)

ODWC_GIS

Recently upgraded our Enterprise deployment in steps from 10.6.1 to 10.8.1 to 11.1.

At 11.1, our Portal Homepage shows no Contents, though the expected folders are still visible.

All of our Services (accessible via the Server Homepage) are still there, and apparently running.

The Portal Administrator Directory's Index Status usually returns 0 for Indexed Items:

And anything I do to remedy the situation is for naught, because running Reindex from Portal Admin results in zeroing the index again.

==============

What I've tried so far:

Visit the ArcGIS Portal Directory, where all of the items I expect to see in the Portal Home are present, if a bit difficult to use.

• Select an item from a folder, Update Item without making any updates.  
  • It appears in the Portal Homepage Contents.  Portal Admin's Index Status shows it in the Index count.
    • Reindex from Portal Admin --> Vanishes from Portal Contents

 

• Select an item from a folder, Update Item without making any updates.  
  • It appears in the Portal Homepage Contents
    • Use ArcGIS Pro to "Overwrite Web Layer" -effectively updating the Service, and interacting with the Portal Content item
      • Reindex from Portal Admin --> Vanishes from Portal Contents

 

• Publish an entirely new Web Layer.  Item appears as an active Service, and has a Portal Content Item.
  • Reindex from Portal Admin --> Vanishes from Portal Contents

 

• Replace the working arcgisportal content items folder with the one added as a template in the installation files.  ...Restarting the Portal Service.
  • Index Status starts at 0; Reindex does nothing to change that status.

 

It is clear to me that Portal Admin's "Reindex" tool isn't working as intended.  I want to know why, and how I can fix it.  Any ideas?

 

Our Server is Federated.

PS: If for any reason our Portal Service is Restarted, some Service Items do not [re]appear in Maps that utilize them... though the Services themselves still run.  My temporary remedy to this is to go through the steps in blue text (above), which appears to resume normal Service services -even after another reindex operation is attempted.

ODWC_GIS

Thanks to those who offered suggestions!

What happened to our Enterprise Deployment may be 'non-standard.'  But I will explain as best as I can understand it myself:

When we were still at the version 10.8.1 state, we transferred the whole deployment from one "box" to another.  This was part of the SQL Server upgrade, and our IT Section's management of data-space; it was done on the advice of an ESRI Technician and with their guidance.  After upgrading the 10.6.1 Deployment to 10.8.1 on the original "box," we created a new 10.8.1 Deployment on the new "box," then joined the new site with the old site while we copied things over [* I'm positive that "copying things over" isn't what we did, but that's what it felt like].   ...Whether or not we left the old "box" running at this time, I'm unsure, but would guess we did (considering the problem we discovered today).

Then we upgraded to 11.1, and attempted to go upon our merry way.  ☹️

What we think happened (posited by an extremely helpful ESRI Tech, Dana A.):
During the time the two Enterprises were joined, some internal confusion happened wherein our deployment confused which site was the primary and which the standby.  While it is possible to unregister the standby machine, you can't unregister the primary.  ...But when we were done with the joint operations and un-joined, the confused configuration attributes still recorded the old "box" as the primary (master node).

How was it discovered:
The arcgisportal\content\items\portal\portal-config.properties file was opened, and compared with the "sharedkey" value in Program Files\ArcGIS\Portal\framework\etc\portal-config.properties and Program Files\ArcGIS\Portal\framework\webapps\arcgis#sharing\WEB-INF\classes\resources\gw-config.properties.

[Honestly, there was a lot more to it, but since even I could tell that something was off here, I'm adding it to this explanation.]

The Fix:
Yeah.... it was a truly dizzying succession of tracking down configuration files from everywhere, opening them up in Notepad, and altering the master node and key files attributes.  Our IT Admin and the ESRI Tech also queried the Log files and watched processes resolve in Task Manager.  While using the template of a "healthy one machine Portal for ArcGIS site" as the goal.

=============

The ESRI Technician who helped us resolve this issue was incredibly professional, proficient, and patient.  She is apparently part of a Technical Support Office in Jordan.  If any of you are having trouble with your Enterprise Deployment, I'd recommend trying to get ahold of someone in that office.