ArcGIS for Server - Error: Failed to start the server machine. Premature end of file

[PF1]

We resolved this issue and wanted to share with the community (Knowledge Base)...

The Problem:

One of our ArcGIS for Server machines was in a stopped state and would not start up.  The only valuable log message we could find in the C:\arcgisserver\logs<machinename>\server directory was:

"Failed to start the server machine. Premature end of file"

This article suggested some sort of corruption in the installation... which we followed, and compared to a working deployment. The specified file (config.xml) had no obvious issues.. it had content and appeared comparable to a working installation.

We found a few log file in the C:\Program Files\ArcGIS\Server\framework\etc\service\logs directory and specifically found the following in the  service.log:

Nov 10, 2016 9:21:05 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
org.xml.sax.SAXParseException; systemId: file: C:/Program Files/ArcGIS/Server/framework/runtime/tomcat/conf/server.xml; lineNumber: 1; columnNumber: 1; Premature end of file.

<STACK TRACE>

and sure enough... the server.xml file was empty (0 bytes)

THE FIX:

1. Copied a server.xml file from an alternate location (a deployment of the same arcgis server version 10.3.1)
2. Modified the port 6443 connector (bottom of file)
   

   <Connector SSLEnabled="true" clientAuth="false" keyAlias="SelfSignedCertificate" keystoreFile="\\server\share\config-store\machines\servername\arcgis.keystore" keystorePass="<OBSCURED> maxHttpHeaderSize="65535" maxPostSize="10485760" maxThreads="150" port="6443" protocol="org.apache.coyote.http11.Http11Protocol" scheme="https" secure="true" sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1" sslProtocol="TLS" /> 

3. Specifically updated the keyAlias, keystoreFile, and keystorePass
4. The keystorePass was a little tricky...  Fortunately we found that the value of password in the \\server\share\config-store\security\super\\super.json file was the same as the keystorepass on other installations... so we put that value into the keystorePass field in our 'hacked' server.xml file.
   
   Note: when the alias, file, or password values were in-correct... the ArcGIS Server would actually startup but only listening on HTTP port 6080. The HTTPS port 6443 (this connector) was non-operational.  This was also the case access HTTPS/443 through the web-adaptor as I assume the web-adaptor sent the back-end request to 6443 not to 6080.  

I hope someone finds this useful if they run into a similar issue.  We will be watching the site for stability, but a quick cursory test shows success and high confidence of this fix.  

[Anonymous User]

Thanks

[ShadyKamel]

Thanks.

just want to add that if you have another machines in a cluster you can just copy the "server.xml" file from another machine in the cluster without any modifications.

[ConradSchaefer__DOIT_]

This was our problem! Space issues must have caused the server.xml to be fouled. It was 0kb. We copied over the server.xml from a healthy machine in our cluster and the web server was restored. We found it at "Program Files\ArcGIS\Server\framework\runtime\tomcat\conf"

[EricO_Neal1]

Thanks P F, that was very helpful.  We had server.xml blanked out on two separate clusters at the same time, but only on the site machine.  Has anyone found the cause of this?  I suspect a windows update or some sort of IT crapware installation...

[pfoppe]

This it us again today, and I forgot the fix... so had to re-discover.  GAR

The root cause was our C:\ hosting the install location ran out of disk space.  So I think the product tried to write to the file and couldn't and ended up writing a 0 byte file....

[KrisAvi]

Had that issue as well. In our case the cause of issues seemed to be machine running out of memory. Lately application service running on same machine had some OutOfMemory errors and also in the ArcGIS logs. I am suspecting it crashed the ArcGIS and tomcat maybe tried to load config to memory, update the ports according to configs and then save it back to file before shutting down, but being out of memory meant 0kb were saved to the file. 

[MoginrajMohandas]

Copying the server.xml from another working server is a good solution here. Glad it worked for you. Remember to re-import any certs you had on the ArcGIS Server machine. It may be also good to do a restart of ArcGIS Server first so that a new keystore gets created for this server, and then re-import any certificates as necessary.

[pfoppe]

Yes, very good points!!

[JohnGibson2]

Excellent article that has "saved my bacon" twice now. First time was after Windows patching, and the second after a random Windows Server crash. In both cases this was with Server 10.3.1 and the config.xml was corrupted. In the second case the log file had the following message : "SEVERE: Parse Fatal Error at line 1 column 1: Content is not allowed in prolog.
org.xml.sax.SAXParseException; systemId: file:/C:/Program%20Files/ArcGIS/Server/framework/runtime/tomcat/conf/server.xml; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog." In this case, both config.xml and web.xml in the same folder were corrupted and stuffed with NUL characters. Replacing these files from another server and reconfiguring config.xml as stated solved the problem. Restarting the ArcGIS Server service then restored the system ok. Thanks for posting this article!