Well, I figured it out. When I was federating server with portal I used the machine name instead of the machine name with the fully qualified domain name (FQDN). For example I put vmname:6443/arcgis instead of vmname.domainname.com:6443/arcgis. This caused portal to look for a certificate for "vmname". Instead, the certificate comes back from server with the vmname.domainname.com name, so they don't match, and an error ensues.
Hope this helps someone out there!