ArcGIS Service started but ArcSOC and javaw processes not started 10.7.1

[SrikanthHemadri]

Hi,

I am having ArcGIS Server 10.7.1 installed which was working fine. I don't know what triggered the issue but suddenly one of my GP Services was not giving response to a Job status request. I restarted the ArcGIS Server service from the system services dialog in my windows server. Only the ArcGIS Server service got started and I cannot see any ArcSOC instances or the Javaw processes running in the processes list.

Because of this I am not able to connect to ArcGIS Server Manager or ArcGIS Admin.

I did some digging and found below log one of the log file present in the path (C:\Program Files\ArcGIS\Server\framework\etc\service\logs).

Mon Sep 21 03:41:24.578 MDT 2020 log4j:WARN No appenders could be found for logger (org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager).
Mon Sep 21 03:41:24.579 MDT 2020 log4j:WARN Please initialize the log4j system properly.
Mon Sep 21 03:41:24.580 MDT 2020 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Mon Sep 21 03:42:54.459 MDT 2020 java.lang.RuntimeException: Unable to connect to Log Service
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger.b(Logger.java:423)
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger.access$300(Logger.java:46)
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger$c.a(Logger$c.java:522)
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger$c.a(Logger$c.java:463)
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger$c.run(Logger$c.java:503)

I repaired the ArcGIS Server installation from Programs and Features in control panel along with running Configure ArcGIS Server Account application but still it did not resolve the issue.

Thanks in Advance

- Srikanth

[ChristoforosKatsaounis]

Hi Srikanth,

What OS is your ArcGIS Server on, and is it a single or multi-machine site?

You mentioned you run the Configure ArcGIS Server account application, but did you doublecheck that the run-as account for ArcGIS Server has read/write permissions minimum at the ArcGIS Server directories, config store, Install directory and Python27 folder? You would want to verify also with your IT/Network admin that there were no Windows updates or any other changes that took place over the weekend.
 
-Christof

[SrikanthHemadri]

Hi Christof,

Thank you very much for responding.

• ArcGIS Server is running on Windows Server 2008 R2 OS and is a Single machine site
• The account which I used for running Configure ArcGIS Server account application is the same one as before which was working fine. It is a local account having administrator privileges and has read/write permissions at the ArcGIS Server directories.
• There was some regular maintenance activity done over the weekend. However along with the server in issue there is another server with similar configurations where the ArcGIS Server installed and is working fine.

One point I missed in the post is that the log level was set to VERBOSE, because of which the disk ran out of space. So I deleted all the log files present in the folder C:\arcgisserver\logs\<server name>\server and cleared upto 30GB of space.

- Srikanth

[ChristoforosKatsaounis]

Hi Srikanth,

Thanks for the details. Does the issue persist after the 30 GB disc space cleanup? 

If yes, we would want to look at the logs from C:\arcgisserver\logs after a restart of the ArcGIS Server Windows service. 

Also, when you mentioned that you're not able to connect to ArcGIS Server Manager or ArcGIS Admin, what is the behavior/error in the UI, and what does the web traffic in the browser show? 

[SrikanthHemadri]

Hi Christof,

Yes the issue persisted even after the space cleanup.

And when I said I'm not able to connect to ArcGIS Server Manager or ArcGIS Admin I am getting error in browser saying "This site can't be reached." Also when I tried from ArcCatalog to connect to the server I am getting the error "Proxy server got bad address from remote server (verify the server is running). (status code 504)."

And Regarding the log files, I have to get them which takes some time. Will upload as soon as I get them. However I was able to see some messages like below in the log files which I found odd.

<Msg time="2020-09-21T21:53:36,282" type="SEVERE" code="7700" source="Server" process="1148" thread="1" methodName="" machine="SERVER.NAME" user="" elapsed="" requestID="">Failed to start web server. Web server failed to start</Msg>

<Msg time="2020-09-22T11:35:24,749" type="WARNING" code="6740" source="Admin" process="1148" thread="1" methodName="" machine="SERVER.NAME" user="" elapsed="" requestID="">The machine 'SERVER.NAME' was not reachable while performing this operation. The server machine will be notified of this change when it comes back online. If ArcGIS Server is going to continue to be unavailable, remove the machine from the site.</Msg>

[ChristoforosKatsaounis · Accepted solution]

Thanks Srikanth. So from the log file errors it seems that the ArcGIS Server's web application server is not starting.

There could be a corruption at the server.xml file in the location below (possibly due to having run out of disk space recently):

C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\conf

The server.xml should have a size > 0 and contain valid xml if its healthy (if you open it in a browser or text editor for example they should show properly formatted xml).

If not the file can be overwritten from an ArcGIS Server of the same version that is working, then restart the ArcGIS Server service. Hope this helps, let me know when you have a chance to try this.

-Christof

[SrikanthHemadri]

Hi Christof,

Sorry for the delay in reply and thank you very much for the solution you provided.

What you mentioned above is exactly what happened. The server.xml is present but of size 0. when I replaced it with the one from a healthy server and restarted everything started working. I would never find it without your help.

-Srikanth

[ChristoforosKatsaounis]

Thank you Srikanth, that's great to hear. Feel free to mark my previous post with the solution as 'Best Answer' so it will be easier for other users to find this if they run into this themselves.

Best,

Christof 

[KannanSundaresan]

Hi All

I am encountering similar issue with ArcGIS on Linux. I just upgraded from 10.7.1 to 10.9.1. upgrade of ArcGIS server component is successful. but after that I cannot connect to server manager through port 6443. I noticed that the Arcsoc process is not running. I checked the server.xml file under server tomcat conf directory. It looks good. It has valid contents. still arcsoc is not starting. restarted the server services multiple times. 

THanks

Kannan

[zrobinson_agolpublic]

Hi Kannan, did you ever resolve the issue? We have the same problem. The server.xml file looks fine but still no ArcSOC processes.