I have a single machine Portal deployment on an Amazon ec2 instance and I have successfully federated ArcGIS Server using the fqdn for the admin url (https://fqdn:6443/arcgis) however; despite this working (and validating) the server will not allow publishing or creation of feature or tile layers. What I found was that the amazon security group needed an inbound rule added to allow ALL traffic which allowed me to access 443 when not on server. Once I was able to get to 443, the admin url with the fqdn would return a cert error since this did not match the certs in place on server. The hosting server would still validate but still could not create hosted tile or feature layers. I used sharing/rest Update Server to change the admin url to the DNS name (https://thistookforevertofigureout.com:6443/arcgis) and this worked.
1. Why did Portal allow me to federate using the fqdn admin url and then successfully validate that server as a valid hosting server when it was not working?
2. Why is the Allow all traffic Rule needed to get to port 443? Is there a more specific rule I can add that will be more secure? https is already added...
Unfortunately, Portal doesn't validate the URLs that the services or admin URL use during federation. So, outside of the port/firewall issues, if the certificate CN doesn't match the URL hostname or SAN doesn't exist for it, publishing will fail since Portal can't talk to Server using the admin URL. In AWS, there are no real "hostnames", so our CF templates use the internal IP address as the hostname.
In terms of the inbound rules, can you describe where the Portal is in relation to the Server? I assume they're on different machines? Are they in the same security group? You definitely shouldn't need to allow all traffic over all ports for Portal and Server to access each other or be able to reach 443 on the machines.
Ok sounds like Portal is not validating full access to the server when you add a Federated Server and then Hosting Server in Portal Settings. The federation makes sense since you can have multiple federated servers however; you would think that if the server cannot function as a Hosting server then it would not let you set this. Long story short, if you cannot access the ArcGIS Server Admin url using https://domain_name:6443/arcgis with no cert errors then don't federate and set hosting server. Portal will allow you to do this but your Portal will not be a functioning hosting server. This is not specific to Amazon and could occur in any deployment using a domain cert.
Single machine deployment, one security group. Without the inbound rule allowing All traffic I cannot get to ipaddress:6443, fqdn:6443, or domain:6443. Added the rule and I'm in...
Firstly I would try adding rules to allow communication via ports 6443 and 6080 and see if that solves the admin federation / content creation issues. Communication tends to happen behind the scenes on these ports and I can see you've enabled 443 for your WA, but an external server might not be able to communicate with the standard admin urls for AGS (i'd probably put 7443 and 7080 in there as well for completeness).
If you want to enable communication from another EC2 within the same VPC (and hopefully on the same domain) then you can add a security group rule to allow all traffic from other EC2 instances which have the same sg applied (rather than via an IP address):
If the above works, you could then refine the allowed ports to simply the standard (6080, 6443, 80, 443) rather than all. This could help you needing to allow all traffic from all IP addresses (which can be a bit of a concern if it's available via a public IP if it's in a public subnet or a load balancer if it's in a private subnet).
Additionally, if all the machines are on the same domain, setting up a CA and installing domain signed certificates to both tomcat (6443) and IIS (443) for your WA (if you're not using the Java WA) solve a lot of other headaches you may have with admin functions working properly.