ArcGIS for Server - Error: Failed to start the server machine. Premature end of file

4126
7
11-10-2016 04:26 PM
Highlighted
by
Occasional Contributor II

We resolved this issue and wanted to share with the community (Knowledge Base)...

The Problem:

One of our ArcGIS for Server machines was in a stopped state and would not start up.  The only valuable log message we could find in the C:\arcgisserver\logs<machinename>\server directory was:

"Failed to start the server machine. Premature end of file"

This article suggested some sort of corruption in the installation... which we followed, and compared to a working deployment. The specified file (config.xml) had no obvious issues.. it had content and appeared comparable to a working installation.

We found a few log file in the C:\Program Files\ArcGIS\Server\framework\etc\service\logs directory and specifically found the following in the  service.log:

Nov 10, 2016 9:21:05 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
org.xml.sax.SAXParseException; systemId: file: C:/Program Files/ArcGIS/Server/framework/runtime/tomcat/conf/server.xml; lineNumber: 1; columnNumber: 1; Premature end of file.

<STACK TRACE>

and sure enough... the server.xml file was empty (0 bytes)

THE FIX:

  1. Copied a server.xml file from an alternate location (a deployment of the same arcgis server version 10.3.1)
  2. Modified the port 6443 connector (bottom of file)
    <Connector SSLEnabled="true" clientAuth="false" keyAlias="SelfSignedCertificate" keystoreFile="\\server\share\config-store\machines\servername\arcgis.keystore" keystorePass="<OBSCURED> maxHttpHeaderSize="65535" maxPostSize="10485760" maxThreads="150" port="6443" protocol="org.apache.coyote.http11.Http11Protocol" scheme="https" secure="true" sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1" sslProtocol="TLS" /> 
  3. Specifically updated the keyAliaskeystoreFile, and keystorePass
  4. The keystorePass was a little tricky...  Fortunately we found that the value of password in the \\server\share\config-store\security\super\\super.json file was the same as the keystorepass on other installations... so we put that value into the keystorePass field in our 'hacked' server.xml file.

    Note: when the alias, file, or password values were in-correct... the ArcGIS Server would actually startup but only listening on HTTP port 6080. The HTTPS port 6443 (this connector) was non-operational.  This was also the case access HTTPS/443 through the web-adaptor as I assume the web-adaptor sent the back-end request to 6443 not to 6080.  

I hope someone finds this useful if they run into a similar issue.  We will be watching the site for stability, but a quick cursory test shows success and high confidence of this fix.  

7 Replies
Highlighted
Occasional Contributor III

Thanks

Reply
0 Kudos
Highlighted
New Contributor II

Thanks.

just want to add that if you have another machines in a cluster you can just copy the "server.xml" file from another machine in the cluster without any modifications.

Highlighted
New Contributor III

Thanks P F, that was very helpful.  We had server.xml blanked out on two separate clusters at the same time, but only on the site machine.  Has anyone found the cause of this?  I suspect a windows update or some sort of IT crapware installation...

Reply
0 Kudos
Highlighted
by MVP
MVP

This it us again today, and I forgot the fix... so had to re-discover.  GAR

The root cause was our C:\ hosting the install location ran out of disk space.  So I think the product tried to write to the file and couldn't and ended up writing a 0 byte file....

Reply
0 Kudos
Highlighted
New Contributor

Had that issue as well. In our case the cause of issues seemed to be machine running out of memory. Lately application service running on same machine had some OutOfMemory errors and also in the ArcGIS logs. I am suspecting it crashed the ArcGIS and tomcat maybe tried to load config to memory, update the ports according to configs and then save it back to file before shutting down, but being out of memory meant 0kb were saved to the file. 

Reply
0 Kudos
Highlighted
New Contributor III

Copying the server.xml from another working server is a good solution here. Glad it worked for you. Remember to re-import any certs you had on the ArcGIS Server machine. It may be also good to do a restart of ArcGIS Server first so that a new keystore gets created for this server, and then re-import any certificates as necessary.

Highlighted
by MVP
MVP

Yes, very good points!!

Reply
0 Kudos