ArcGIS Service started but ArcSOC and javaw processes not started 10.7.1

1218
7
Jump to solution
09-21-2020 07:58 AM
SrikanthHemadri
New Contributor II

Hi,

I am having ArcGIS Server 10.7.1 installed which was working fine. I don't know what triggered the issue but suddenly one of my GP Services was not giving response to a Job status request. I restarted the ArcGIS Server service from the system services dialog in my windows server. Only the ArcGIS Server service got started and I cannot see any ArcSOC instances or the Javaw processes running in the processes list.

Because of this I am not able to connect to ArcGIS Server Manager or ArcGIS Admin.

I did some digging and found below log one of the log file present in the path (C:\Program Files\ArcGIS\Server\framework\etc\service\logs).

Mon Sep 21 03:41:24.578 MDT 2020 log4j:WARN No appenders could be found for logger (org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager).
Mon Sep 21 03:41:24.579 MDT 2020 log4j:WARN Please initialize the log4j system properly.
Mon Sep 21 03:41:24.580 MDT 2020 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Mon Sep 21 03:42:54.459 MDT 2020 java.lang.RuntimeException: Unable to connect to Log Service
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger.b(Logger.java:423)
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger.access$300(Logger.java:46)
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger$c.a(Logger$c.java:522)
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger$c.a(Logger$c.java:463)
Mon Sep 21 03:42:54.459 MDT 2020 at com.esri.arcgis.discovery.logging.Logger$c.run(Logger$c.java:503)

I repaired the ArcGIS Server installation from Programs and Features in control panel along with running Configure ArcGIS Server Account application but still it did not resolve the issue.

Thanks in Advance

- Srikanth

0 Kudos
1 Solution

Accepted Solutions
ChristoforosKatsaounis
Esri Contributor

Thanks Srikanth. So from the log file errors it seems that the ArcGIS Server's web application server is not starting.

There could be a corruption at the server.xml file in the location below (possibly due to having run out of disk space recently):

C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\conf

The server.xml should have a size > 0 and contain valid xml if its healthy (if you open it in a browser or text editor for example they should show properly formatted xml).

If not the file can be overwritten from an ArcGIS Server of the same version that is working, then restart the ArcGIS Server service. Hope this helps, let me know when you have a chance to try this.

-Christof

View solution in original post

7 Replies
ChristoforosKatsaounis
Esri Contributor

Hi Srikanth,

What OS is your ArcGIS Server on, and is it a single or multi-machine site?

You mentioned you run the Configure ArcGIS Server account application, but did you doublecheck that the run-as account for ArcGIS Server has read/write permissions minimum at the ArcGIS Server directories, config store, Install directory and Python27 folder? You would want to verify also with your IT/Network admin that there were no Windows updates or any other changes that took place over the weekend.
 
-Christof

0 Kudos
SrikanthHemadri
New Contributor II

Hi Christof,

Thank you very much for responding.

  • ArcGIS Server is running on Windows Server 2008 R2 OS and is a Single machine site
  • The account which I used for running Configure ArcGIS Server account application is the same one as before which was working fine. It is a local account having administrator privileges and has read/write permissions at the ArcGIS Server directories.
  • There was some regular maintenance activity done over the weekend. However along with the server in issue there is another server with similar configurations where the ArcGIS Server installed and is working fine.

One point I missed in the post is that the log level was set to VERBOSE, because of which the disk ran out of space. So I deleted all the log files present in the folder C:\arcgisserver\logs\<server name>\server and cleared upto 30GB of space.

- Srikanth

0 Kudos
ChristoforosKatsaounis
Esri Contributor

Hi Srikanth,

Thanks for the details. Does the issue persist after the 30 GB disc space cleanup? 

If yes, we would want to look at the logs from C:\arcgisserver\logs after a restart of the ArcGIS Server Windows service. 

Also, when you mentioned that you're not able to connect to ArcGIS Server Manager or ArcGIS Admin, what is the behavior/error in the UI, and what does the web traffic in the browser show? 

0 Kudos
SrikanthHemadri
New Contributor II

Hi Christof,

Yes the issue persisted even after the space cleanup.

And when I said I'm not able to connect to ArcGIS Server Manager or ArcGIS Admin I am getting error in browser saying "This site can't be reached." Also when I tried from ArcCatalog to connect to the server I am getting the error "Proxy server got bad address from remote server (verify the server is running). (status code 504)."

And Regarding the log files, I have to get them which takes some time. Will upload as soon as I get them. However I was able to see some messages like below in the log files which I found odd.

<Msg time="2020-09-21T21:53:36,282" type="SEVERE" code="7700" source="Server" process="1148" thread="1" methodName="" machine="SERVER.NAME" user="" elapsed="" requestID="">Failed to start web server. Web server failed to start</Msg>
<Msg time="2020-09-22T11:35:24,749" type="WARNING" code="6740" source="Admin" process="1148" thread="1" methodName="" machine="SERVER.NAME" user="" elapsed="" requestID="">The machine 'SERVER.NAME' was not reachable while performing this operation. The server machine will be notified of this change when it comes back online. If ArcGIS Server is going to continue to be unavailable, remove the machine from the site.</Msg>
0 Kudos
ChristoforosKatsaounis
Esri Contributor

Thanks Srikanth. So from the log file errors it seems that the ArcGIS Server's web application server is not starting.

There could be a corruption at the server.xml file in the location below (possibly due to having run out of disk space recently):

C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\conf

The server.xml should have a size > 0 and contain valid xml if its healthy (if you open it in a browser or text editor for example they should show properly formatted xml).

If not the file can be overwritten from an ArcGIS Server of the same version that is working, then restart the ArcGIS Server service. Hope this helps, let me know when you have a chance to try this.

-Christof

View solution in original post

SrikanthHemadri
New Contributor II

Hi Christof,

Sorry for the delay in reply and thank you very much for the solution you provided.

What you mentioned above is exactly what happened. The server.xml is present but of size 0. when I replaced it with the one from a healthy server and restarted everything started working. I would never find it without your help.

 

-Srikanth

0 Kudos
ChristoforosKatsaounis
Esri Contributor

Thank you Srikanth, that's great to hear. Feel free to mark my previous post with the solution as 'Best Answer' so it will be easier for other users to find this if they run into this themselves.

Best,

Christof 

0 Kudos