ArcGIS for Server - Error: Failed to start the server machine. Premature end of file

8640
9
11-10-2016 04:26 PM
PF1
by
Occasional Contributor II

We resolved this issue and wanted to share with the community (Knowledge Base)...

The Problem:

One of our ArcGIS for Server machines was in a stopped state and would not start up.  The only valuable log message we could find in the C:\arcgisserver\logs<machinename>\server directory was:

"Failed to start the server machine. Premature end of file"

This article suggested some sort of corruption in the installation... which we followed, and compared to a working deployment. The specified file (config.xml) had no obvious issues.. it had content and appeared comparable to a working installation.

We found a few log file in the C:\Program Files\ArcGIS\Server\framework\etc\service\logs directory and specifically found the following in the  service.log:

Nov 10, 2016 9:21:05 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
org.xml.sax.SAXParseException; systemId: file: C:/Program Files/ArcGIS/Server/framework/runtime/tomcat/conf/server.xml; lineNumber: 1; columnNumber: 1; Premature end of file.

<STACK TRACE>

and sure enough... the server.xml file was empty (0 bytes)

THE FIX:

  1. Copied a server.xml file from an alternate location (a deployment of the same arcgis server version 10.3.1)
  2. Modified the port 6443 connector (bottom of file)
    <Connector SSLEnabled="true" clientAuth="false" keyAlias="SelfSignedCertificate" keystoreFile="\\server\share\config-store\machines\servername\arcgis.keystore" keystorePass="<OBSCURED> maxHttpHeaderSize="65535" maxPostSize="10485760" maxThreads="150" port="6443" protocol="org.apache.coyote.http11.Http11Protocol" scheme="https" secure="true" sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1" sslProtocol="TLS" /> 
  3. Specifically updated the keyAliaskeystoreFile, and keystorePass
  4. The keystorePass was a little tricky...  Fortunately we found that the value of password in the \\server\share\config-store\security\super\\super.json file was the same as the keystorepass on other installations... so we put that value into the keystorePass field in our 'hacked' server.xml file.

    Note: when the alias, file, or password values were in-correct... the ArcGIS Server would actually startup but only listening on HTTP port 6080. The HTTPS port 6443 (this connector) was non-operational.  This was also the case access HTTPS/443 through the web-adaptor as I assume the web-adaptor sent the back-end request to 6443 not to 6080.  

I hope someone finds this useful if they run into a similar issue.  We will be watching the site for stability, but a quick cursory test shows success and high confidence of this fix.  

9 Replies
by Anonymous User
Not applicable

Thanks

0 Kudos
ShadyKamel
New Contributor II

Thanks.

just want to add that if you have another machines in a cluster you can just copy the "server.xml" file from another machine in the cluster without any modifications.

ConradSchaefer__DOIT_
New Contributor III

This was our problem! Space issues must have caused the server.xml to be fouled. It was 0kb. We copied over the server.xml from a healthy machine in our cluster and the web server was restored. We found it at "Program Files\ArcGIS\Server\framework\runtime\tomcat\conf"

0 Kudos
EricO_Neal1
New Contributor III

Thanks P F, that was very helpful.  We had server.xml blanked out on two separate clusters at the same time, but only on the site machine.  Has anyone found the cause of this?  I suspect a windows update or some sort of IT crapware installation...

0 Kudos
pfoppe
by MVP
MVP

This it us again today, and I forgot the fix... so had to re-discover.  GAR

The root cause was our C:\ hosting the install location ran out of disk space.  So I think the product tried to write to the file and couldn't and ended up writing a 0 byte file....

0 Kudos
KrisAvi
New Contributor

Had that issue as well. In our case the cause of issues seemed to be machine running out of memory. Lately application service running on same machine had some OutOfMemory errors and also in the ArcGIS logs. I am suspecting it crashed the ArcGIS and tomcat maybe tried to load config to memory, update the ports according to configs and then save it back to file before shutting down, but being out of memory meant 0kb were saved to the file. 

0 Kudos
MoginrajMohandas
New Contributor III

Copying the server.xml from another working server is a good solution here. Glad it worked for you. Remember to re-import any certs you had on the ArcGIS Server machine. It may be also good to do a restart of ArcGIS Server first so that a new keystore gets created for this server, and then re-import any certificates as necessary.

pfoppe
by MVP
MVP

Yes, very good points!!

0 Kudos
JohnGibson2
New Contributor II

Excellent article that has "saved my bacon" twice now. First time was after Windows patching, and the second after a random Windows Server crash. In both cases this was with Server 10.3.1 and the config.xml was corrupted. In the second case the log file had the following message : "SEVERE: Parse Fatal Error at line 1 column 1: Content is not allowed in prolog.
org.xml.sax.SAXParseException; systemId: file:/C:/Program%20Files/ArcGIS/Server/framework/runtime/tomcat/conf/server.xml; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog." In this case, both config.xml and web.xml in the same folder were corrupted and stuffed with NUL characters. Replacing these files from another server and reconfiguring config.xml as stated solved the problem. Restarting the ArcGIS Server service then restored the system ok. Thanks for posting this article!

0 Kudos