AnsweredAssumed Answered

ArcGIS Server 10.5 server starting but services failing to start

Question asked by yucel_tepekoy on Mar 30, 2017
Latest reply on Jan 11, 2018 by randall_williams-esristaff

ArcGIS Enterprise 10.5 installed on OEL 7.2. The serverdiag seems to be OK with the warning of core services not starting. The service-error.log file indicates error in Java below. The service.log  seems OK. But the problem is none of the services are starting. We just recognized that the ArcGIS server wasn't working probably for 2-3 days today, so we can not trace back to what the problem might be. I also changed the SERVER TMP in usr/init_user_param.sh but that didn't work also after a server stop and start.

 

Any suggestions from the below logs of why the services might not be starting?

 

Service-error.log file contents:

Mar 30, 2017 2:50:23 PM ClientCommunicatorAdmin restart

WARNING: Failed to restart: java.rmi.NoSuchObjectException: no such object in table

Mar 30, 2017 2:50:23 PM ClientCommunicatorAdmin restart

WARNING: Failed to restart: java.io.IOException: Failed to get a RMI stub: javax.naming.NameNotFoundException: jmxrmi

java.net.ConnectException: Connection refused

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

        at java.net.Socket.connect(Socket.java:589)

        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)

        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)

        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)

        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)

        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)

        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:136)

        at com.esri.arcgis.discovery.admin.util.HttpUtil.postToURL(HttpUtil.java:405)

        at com.esri.arcgis.discovery.admin.util.HttpUtil.postPropsToURL(HttpUtil.java:329)

        at com.esri.arcgis.discovery.admin.system.handlers.RestConfigManager$a.call(RestConfigManager$a.java:710)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

 

 

Service.log file contents (last 100 lines)

 

 

JMX Closed

WARNING: Could not connect to Log service. Writing log message to console.

<Msg time="2017-03-30T14:50:23,943" type="WARNING" code="6740" source="Admin" process="14311" thread="1" methodName="" machine="" user="" elapsed="">The machine '10.0.87.21' was not reachable while performing this operation. The server machine will be notified of this change when it comes back online. If ArcGIS Server is going to continue to be unavailable, remove the machine from the site.</Msg>

Start invoked

Thu Mar 30 14:50:33 AZT 2017:Start NodeAgent.start().

Thu Mar 30 14:50:33 AZT 2017:Loading all observers.

arcgis_cloud_platform=null

arcgis_cloud_platform=null

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'LogServiceObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'ServerUninstallObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'ServerConfigObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'Server upgrader'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'.

Thu Mar 30 14:50:33 AZT 2017:Regular startup of Server.

Thu Mar 30 14:50:33 AZT 2017:Starting RMI connector for NodeAgent.

Thu Mar 30 14:50:34 AZT 2017:Invoking beforeStart() for all observers.

Thu Mar 30 14:50:34 AZT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi17983:9

Thu Mar 30 14:50:34 AZT 2017:Registering and starting log service on TEKUISPARCGIS01.EMDKDEDRX.GOV.AZ.

Thu Mar 30 14:50:34 AZT 2017:Registering NodeAgent as JMX bean.

Thu Mar 30 14:50:34 AZT 2017:Loading all plugins.

arcgis_cloud_platform=null

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ProcessMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'DirectoryCleaner'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'UlimitMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'LogFileLimiter'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ConfigSynchronizer'.

Thu Mar 30 14:50:34 AZT 2017:Loading plugin.. com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor

Thu Mar 30 14:50:34 AZT 2017:Props {PeriodicInterval=1, NodeAgentCustomPlugin=false}

Thu Mar 30 14:50:34 AZT 2017:Loaded plugin com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ServiceInstanceMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'PlatformServicesPluginObserver'.

Thu Mar 30 14:50:34 AZT 2017:Starting all plugins.arcgis

StartRMID - arg: -J-Djava.security.policy=file:////home/arcgis/server/framework/etc/rmid.policy

StartRMID - arg: -J-Dsun.rmi.activation.execPolicy=com.esri.arcgis.discovery.servicelib.impl.RMIDPolicyChecker

StartRMID - arg: -J-Djava.class.path=/home/arcgis/server/framework/lib/server/arcgis-servicelib.jar:/home/arcgis/server/framework/lib/server/arcgis-common.jar

StartRMID - arg: -J-Djava.rmi.server.useCodebaseOnly=true

StartRMID - arg: -port

StartRMID - arg: 1098

 

 

Using GERONIMO_HOME:   /home/arcgis/server/geronimo

Using GERONIMO_TMPDIR: var/temp

Using ARCGIS_JAVA_HOME:        /home/arcgis/server//framework/runtime/jre

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

Booting Geronimo Kernel (in Java 1.8.0_101)...

Starting Geronimo Application Server v2.2.2-SNAPSHOT

[****************************************] 100%   2s Startup complete          

  Listening on Ports:

    4000 0.0.0.0 OpenEJB Daemon

    4001 0.0.0.0 JMX Remoting Connector

    4002 0.0.0.0 RMI Naming

    4003 0.0.0.0 Derby Connector

 

  Started Application Modules:

    JAR: com.esri.arcgis.services/GeronimoSingleton/1.0/car

    JAR: com.esri.arcgis.services/ServiceCatalog/1.0/car

    JAR: com.esri.arcgis.services/System.CachingToolsEx.GPServer/1.0/car

    JAR: com.esri.arcgis.services/System.DynamicMappingHost.MapServer/1.0/car

    JAR: com.esri.arcgis.services/System.PublishingToolsEx.GPServer/1.0/car

    JAR: org.apache.geronimo.configs/mejb/2.2.2-SNAPSHOT/car

    RAR: org.apache.geronimo.configs/system-database/2.2.2-SNAPSHOT/car

 

Geronimo Application Server started

system.reg is not a valid registry file

userdef.reg:4: Mismatched architecture '#arch=win64'

userdef.reg is not a valid registry file

user.reg:4: Mismatched architecture '#arch=win64'

user.reg is not a valid registry file

wine: '/home/arcgis/server/framework/runtime/.wine' is a 32-bit installation, it cannot support 64-bit applications.

Outcomes