Hi Holger,
What you need to enter is what you see after a right click on the database in ArcCatalog and choose "Save connection". You can then open the "Connection Properties" from the connection file under the "Database Connections" node.
In your case, you seem to be missing the correct xpression for the service, try
sde:sqlserver:[machinename]\sqlexpress
Best regards,
Matthias
ESRI Switzerland