I would suggest attempting to install ArcGIS Server 10.4 using a local account instead of a domain account. After installing ArcGIS Server 10.4 you can open a program called "Configure ArcGIS Server Account" where you can then attempt to change the ArcGIS Server Account to a domain account.
You should also go to Control Panel > System and Security > System and make sure the computer is actually connected to a domain. As you can see below, my computer is connected to the esri.com domain. If the computer is not connected to a domain, then I imagine you will not be able to configure ArcGIS Server 10.4 with a domain account for the ArcGIS Server Account.