ArcGIS service Stops immediately after it starts

[BledarBirbo1]

Hi.

Yesterday the pc that was running ArcGIS Server 10.2.2 shutted down unexpectedly and since then the ArcGIS service Stops immediately after it starts. ( Even manually or Automatically )

Here is what the log file says

<Msg time="2016-01-07T11:46:11,915" type="INFO" code="7702" source="Admin" process="3796" thread="1" methodName="" machine="MACHINE1.COM" user="" elapsed="">Web server started successfully.</Msg>

<Msg time="2016-01-07T11:46:16,88" type="SEVERE" code="6615" source="Admin" process="3796" thread="1" methodName="" machine="MACHINE1.COM" user="" elapsed="">Unable to retrieve 'Permissions' resource information. Content is not allowed in prolog.</Msg>

<Msg time="2016-01-07T11:46:16,89" type="DEBUG" code="9999" source="Admin" process="3796" thread="1" methodName="" machine="MACHINE1.COM" user="" elapsed="">com.esri.arcgis.discovery.admin.security.AGSSecurityException: Content is not allowed in prolog.

    at com.esri.arcgis.discovery.admin.security.impl.FSPermissionManager.initialize(FSPermissionManager.java:85)

    at com.esri.arcgis.discovery.admin.SiteManager.getPermissionManager(SiteManager.java:1717)

    at com.esri.arcgis.discovery.admin.SiteManager.getPermissionManager(SiteManager.java:1691)

    at com.esri.arcgis.discovery.admin.SiteManager.a(SiteManager.java:256)

    at com.esri.arcgis.discovery.admin.SiteManager.&lt;init&gt;(SiteManager.java:237)

    at com.esri.arcgis.discovery.admin.SiteManager.&lt;init&gt;(SiteManager.java:154)

    at com.esri.arcgis.discovery.admin.SiteManager$i_.&lt;clinit&gt;(SiteManager$i_.java:356)

    at com.esri.arcgis.discovery.admin.SiteManager.getInstance(SiteManager.java:367)

    at com.esri.arcgis.discovery.admin.plugins.DirectoryCleaner$b_.run(DirectoryCleaner$b_.java:167)

Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.

    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)

    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)

    at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)

    at com.esri.arcgis.discovery.util.XMLUtil.openXMLDocument(XMLUtil.java:112)

    at com.esri.arcgis.discovery.admin.security.impl.FSPermissionManager.c(FSPermissionManager.java:913)

    at com.esri.arcgis.discovery.admin.security.impl.FSPermissionManager.b(FSPermissionManager.java:823)

    at com.esri.arcgis.discovery.admin.security.impl.FSPermissionManager.a(FSPermissionManager.java:791)

    at com.esri.arcgis.discovery.admin.security.impl.FSPermissionManager.initialize(FSPermissionManager.java:81)

    ... 8 more

</Msg>

Apparently there should be a corrupted XML file somewhere but we couldn't find which one.

Any idea on how to get the exact XML filename that is causing the problem ?

Thanks.

[RebeccaStrauch__GISP]

Any chance that the password for the services that runs your ArcGIS Server services (that is, the OS services) has changes or expired?  I know it is recommended to run the services with a system-accuont with a (complex) password that doesn't expire, if you network allows it.

Another place to check for password changes or expired is in he ArcGIS Service management security tab....if the user used for your user/security access has a bad password.  For example, I initially set up our active-directory access to my user/pass.  I am the admin so this worked great, until 90-days passed and I had to change y password...that broke things (see note about system account above)

One other thing to check, if by chance the AGS was registered using an Esri Developer Network (EDN) license, those will expire each year and need to be reauthorized (to be used for testing and development only, not for production of course).  This can break ArcSDE databases too.

I mention these three items because these are things that can suddenly "break" AGS, without anything else really being in error.  Simple thins to check.

[BledarBirbo1]

Hi Rebecca.

Unfortunately we could not fix this issue.

After spending 4 hours in calls with ESRI support we ended up reinstalling the ArcGIS server.

Thanks for your response.

[RebeccaStrauch__GISP]

Sometimes that is the best/only option.  The main things is you did get it up and running again.  I know I've run into that issues before but can't remember the details of how I resolved it...may have been a reinstall too.

Since there is no real "fix", (other that a reinstall), you can mark this question as "assumed answered" so the thread can be closed.

[ChrisSmith7]

Bledar,

Just a heads-up in case you weren't aware, even if you just have a single instance of ArcGIS Server, you can run a passive failover node... you just have to e-mail and request authorization from Esri. That way, worst-case scenario, you can keep a mirror copy and failover while you rebuild/reinstall the failed machine.

[BledarBirbo1]

Thanks Chris.

I will do some more research about this and probably implement this method.

[VinceAngelo]

You should probably run a filesystem check on your install disk.  Sometimes disk content failure is due to random electromagnetic effects, but sometimes it's due to physical damage to the disk platter.  I once had a client lose 48Tb of raster data (the better part of 9 months of loading effort) due to a firmware failure in the disk array controller (they had a backup array, but that was corrupted by mirroring the production volume).

Systematic backups (actually, verified systematic backups) aren't just a good idea.

- V

[BledarBirbo1]

I think the problem was because the PC was shutted down during some XML file write process and thus left the file corrupted.