What type of SSL Certificate are you using? I believe self-signed will give you problems beyond the initial installation.
It does seem that your problem is Web Adaptor related.
When you access Server Manager and REST endpoints via URL, is that via the web adaptor address or via port?
I think I would try (you are using FQDN?)
1. Verify that Portal, ArcGIS Server and Web Adaptors are all 10.4.1
(was this an upgrade or a clean install? Same box? Did you install Data Store? Is AGS Federated ?
1.b Verify the Services are running
1.c Check the Server logs
1.d Check the Portal logs via PortalAdmin
1.e Verify IIS is correctly setup (you've bound the SSL (not self-signed) to port 443 in IIS?)
2. Check the ArcGIS Server Web Adaptor is correct and functioning as expected.
3. The Portal web adaptor looks correct above but it seems that something is off.
I would try uninstalling it and reinstalling.
If that fails, I would probably uninstall both AGS and Portal Web Adaptors and try reinstalling.
Still dead:
4. Put in a support ticket? or uninstall and reinstall Portal and it's Web Adaptor then a support ticket if no luck.
FYI - I use the Chef cookbooks for building up an all in one box.
It has made my life much easier and the installations are consistent.
EDIT: couple of threads that might be of use say:
Clean cookies, try different browser,
Portal Users "Sign In" Problem
Check the https://YourPortalFQDN/portal/portaladmin/system/webadaptors
What's the status show there? Ditto for AGS.