Please provide the version of ArcGIS Server, ArcSDE, and SQL Server that you are using. It would also be helpful to provide a screenshot of the error you are getting. Does the SDE connection file work when trying to simply view the data via ArcCatalog? In other words, is the problem only occurring when you attempt to register the database with ArcGIS Server?