Catalog pane cannot see any portal items

[DanNarsavage_IDWR]

The Problem:
When visiting our portal in a web browser, we can see and manipulate all our items as we expect.  We can download a PITEMX file from the portal and open that to see the layer in a map in ArcGIS Pro.  But when we connect to the same portal in ArcGIS Pro and look at the "Portal" tab in the "Catalog" pane, we see zero items.  We can see groups, but no items that are within those groups.  We also cannot publish a geoprocessing tool through ArcGIS Pro.  See the log below. 

This is new as of a week or two ago. We had no problems for the couple months we've had this portal prior to that.  We don't know what changed.  Has anyone else experienced this?  Does anyone have troubleshooting tips for this?  Or workarounds?  Or patches?  Or chemical dependencies?

The Deets:

• ArcGIS Pro is licensed via named users in AGOL
• The connection from Pro to Portal happens successfully--we can see our profiles, the proper portal URL, and groups
• Our portal is running on-prem ArcGIS Enterprise v10.8.1.  Patches are applied monthly. 
• We've experienced this behavior on at least three machines at ArcGIS Pro v3.1.0, 3.1.1, 3.1.2, and 2.8

The Screen Shots:

Proof that we have items in our portal and that we can see them in a browser:

Nothing in the catalog when we view our org's items:

But we can see groups!

The Publishing Log:

Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:33 Status: InProgress StatusMessage: Successfully wrote sharing info file
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:33 Status: InProgress StatusMessage: Loading manifest
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:33 Status: InProgress StatusMessage: Creating thumbnail from map
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:33 Status: Pending StatusMessage: Sharing process started
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:35 Status: InProgress StatusMessage: Sharing process started
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:35 Status: InProgress StatusMessage: Loading manifest
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:35 Status: InProgress StatusMessage: Creating thumbnail from map
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:35 Status: InProgress StatusMessage: Publishing process started
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:35 Status: InProgress StatusMessage: Preparing data references and staging web tool
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:38 Status: InProgress StatusMessage: Using ArcGIS Version 12.6 for staging (Source for version: GP Staging Parameters)
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:38 Status: InProgress StatusMessage: Stage Tool initialized
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:38 Status: InProgress StatusMessage: Consolidate data
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:38 Status: InProgress StatusMessage: Staging GP service
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:49 Status: InProgress StatusMessage: Compressing package into SD file
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:49 Status: InProgress StatusMessage: Staging successful
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:49 Status: InProgress StatusMessage: Uploading service definition
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:50 Status: InProgress StatusMessage: Publishing tool initialized
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:50 Status: InProgress StatusMessage: Publishing web layer (AGS)
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:52 Status: InProgress StatusMessage: Publishing tool execution failed
Layout 307 WebTool Idaho Department of Water Resources Development https://our.portal.url/hosting Username@our.domain 2023-06-21 14:18:54 Status: Failed ErrorMessage: Failed to publish web tool

[Robert_LeClair]

Question for you - what licensing model are you using?  Single/Concurrent/Named User?

[DanNarsavage_IDWR]

Thanks for the question!  I've added this info in the "Deets" section.  We get our Pro licenses via named users in AGOL.

[Robert_LeClair]

Ah right you are!  Apologies for the oversight.  Will continue to look into this...

[DanNarsavage_IDWR]

It wasn't an oversight.  I added it to the Deets section *after* you asked. 🙂

[DanNarsavage_IDWR · Accepted solution]

This was related to DNS/routing issues.  This was penned by a member of our IT hosting team . . . 

"Turns out... when you have custom DNS with multiple zones that apply to different people within the same organization that will then in turn route them through different channels in the F5... they get different polices and well.......stuff breaks."