When federating with a multimachine site or highly available ArcGIS Server, use the "Web Adaptor or load balancer URL" in the Administration URL field instead of the https://machinename.com:6443/arcgis.
The Administration URL must be a URL that the Portal can use to communicate with ALL Servers in the SITE, even when one of them is unavailable. If you use a Web Adaptor for this URL, ensure that you have enabled administrative access to the server through the Web Adaptor.
This is mentioned as part of the "Note" in the documentation (10.5.x)
*Also one thing to point out, which is also mentioned in the documentation, as of version 10.5.1, a Portal can federate with a 10.5 server as long as it is not set as a Hosting Server, GeoAnalytics Server or Raster Analysis Server.
Federate an ArcGIS Server site with your portal: