ArcGIS Server 10.5 server starting but services failing to start

6368
3
03-30-2017 03:59 AM
YucelTepekoy1
New Contributor

ArcGIS Enterprise 10.5 installed on OEL 7.2. The serverdiag seems to be OK with the warning of core services not starting. The service-error.log file indicates error in Java below. The service.log  seems OK. But the problem is none of the services are starting. We just recognized that the ArcGIS server wasn't working probably for 2-3 days today, so we can not trace back to what the problem might be. I also changed the SERVER TMP in usr/init_user_param.sh but that didn't work also after a server stop and start.

Any suggestions from the below logs of why the services might not be starting?

Service-error.log file contents:

Mar 30, 2017 2:50:23 PM ClientCommunicatorAdmin restart

WARNING: Failed to restart: java.rmi.NoSuchObjectException: no such object in table

Mar 30, 2017 2:50:23 PM ClientCommunicatorAdmin restart

WARNING: Failed to restart: java.io.IOException: Failed to get a RMI stub: javax.naming.NameNotFoundException: jmxrmi

java.net.ConnectException: Connection refused

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

        at java.net.Socket.connect(Socket.java:589)

        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)

        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)

        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)

        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)

        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)

        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:136)

        at com.esri.arcgis.discovery.admin.util.HttpUtil.postToURL(HttpUtil.java:405)

        at com.esri.arcgis.discovery.admin.util.HttpUtil.postPropsToURL(HttpUtil.java:329)

        at com.esri.arcgis.discovery.admin.system.handlers.RestConfigManager$a.call(RestConfigManager$a.java:710)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Service.log file contents (last 100 lines)

 

JMX Closed

WARNING: Could not connect to Log service. Writing log message to console.

<Msg time="2017-03-30T14:50:23,943" type="WARNING" code="6740" source="Admin" process="14311" thread="1" methodName="" machine="" user="" elapsed="">The machine '10.0.87.21' was not reachable while performing this operation. The server machine will be notified of this change when it comes back online. If ArcGIS Server is going to continue to be unavailable, remove the machine from the site.</Msg>

Start invoked

Thu Mar 30 14:50:33 AZT 2017:Start NodeAgent.start().

Thu Mar 30 14:50:33 AZT 2017:Loading all observers.

arcgis_cloud_platform=null

arcgis_cloud_platform=null

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'LogServiceObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'ServerUninstallObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'ServerConfigObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'Server upgrader'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'.

Thu Mar 30 14:50:33 AZT 2017:Regular startup of Server.

Thu Mar 30 14:50:33 AZT 2017:Starting RMI connector for NodeAgent.

Thu Mar 30 14:50:34 AZT 2017:Invoking beforeStart() for all observers.

Thu Mar 30 14:50:34 AZT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi17983:9

Thu Mar 30 14:50:34 AZT 2017:Registering and starting log service on TEKUISPARCGIS01.EMDKDEDRX.GOV.AZ.

Thu Mar 30 14:50:34 AZT 2017:Registering NodeAgent as JMX bean.

Thu Mar 30 14:50:34 AZT 2017:Loading all plugins.

arcgis_cloud_platform=null

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ProcessMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'DirectoryCleaner'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'UlimitMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'LogFileLimiter'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ConfigSynchronizer'.

Thu Mar 30 14:50:34 AZT 2017:Loading plugin.. com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor

Thu Mar 30 14:50:34 AZT 2017:Props {PeriodicInterval=1, NodeAgentCustomPlugin=false}

Thu Mar 30 14:50:34 AZT 2017:Loaded plugin com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ServiceInstanceMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'PlatformServicesPluginObserver'.

Thu Mar 30 14:50:34 AZT 2017:Starting all plugins.arcgis

StartRMID - arg: -J-Djava.security.policy=file:////home/arcgis/server/framework/etc/rmid.policy

StartRMID - arg: -J-Dsun.rmi.activation.execPolicy=com.esri.arcgis.discovery.servicelib.impl.RMIDPolicyChecker

StartRMID - arg: -J-Djava.class.path=/home/arcgis/server/framework/lib/server/arcgis-servicelib.jar:/home/arcgis/server/framework/lib/server/arcgis-common.jar

StartRMID - arg: -J-Djava.rmi.server.useCodebaseOnly=true

StartRMID - arg: -port

StartRMID - arg: 1098

Using GERONIMO_HOME:   /home/arcgis/server/geronimo

Using GERONIMO_TMPDIR: var/temp

Using ARCGIS_JAVA_HOME:        /home/arcgis/server//framework/runtime/jre

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

Booting Geronimo Kernel (in Java 1.8.0_101)...

Starting Geronimo Application Server v2.2.2-SNAPSHOT

[****************************************] 100%   2s Startup complete          

  Listening on Ports:

    4000 0.0.0.0 OpenEJB Daemon

    4001 0.0.0.0 JMX Remoting Connector

    4002 0.0.0.0 RMI Naming

    4003 0.0.0.0 Derby Connector

  Started Application Modules:

    JAR: com.esri.arcgis.services/GeronimoSingleton/1.0/car

    JAR: com.esri.arcgis.services/ServiceCatalog/1.0/car

    JAR: com.esri.arcgis.services/System.CachingToolsEx.GPServer/1.0/car

    JAR: com.esri.arcgis.services/System.DynamicMappingHost.MapServer/1.0/car

    JAR: com.esri.arcgis.services/System.PublishingToolsEx.GPServer/1.0/car

    JAR: org.apache.geronimo.configs/mejb/2.2.2-SNAPSHOT/car

    RAR: org.apache.geronimo.configs/system-database/2.2.2-SNAPSHOT/car

Geronimo Application Server started

system.reg is not a valid registry file

userdef.reg:4: Mismatched architecture '#arch=win64'

userdef.reg is not a valid registry file

user.reg:4: Mismatched architecture '#arch=win64'

user.reg is not a valid registry file

wine: '/home/arcgis/server/framework/runtime/.wine' is a 32-bit installation, it cannot support 64-bit applications.

3 Replies
ApurvDanke
Occasional Contributor

Hi All,

I am getting a similar error in the service.log file(as given below). The ArcGIS Server version is 10.4.1 for Windows. The operating system is Windows Server 2012 R2. Due to this we're getting error while making a connection in ArcCatalog to administer ArcGIS Server. That error message is -> 

Tue Dec 19 06:55:00 GMT 2017:End NodeAgent.stop(true).
Start invoked
Tue Dec 19 09:40:01 GMT 2017:Start NodeAgent.start().
Tue Dec 19 09:40:01 GMT 2017:Loading all observers.
arcgis_cloud_platform=null
arcgis_cloud_platform=null
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'LogServiceObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'ServerUninstallObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'ServerConfigObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'Server upgrader'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'.
Tue Dec 19 09:40:03 GMT 2017:Regular startup of Server.
Tue Dec 19 09:40:03 GMT 2017:Starting RMI connector for NodeAgent.
Tue Dec 19 09:40:04 GMT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]20948:10
Tue Dec 19 09:40:04 GMT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi20948:12
Tue Dec 19 09:40:04 GMT 2017:Invoking beforeStart() for all observers.
Tue Dec 19 09:40:05 GMT 2017:Registering and starting log service on <arcgis_server_hostname>.
Tue Dec 19 09:40:07 GMT 2017:Registering NodeAgent as JMX bean.
Tue Dec 19 09:40:07 GMT 2017:Loading all plugins.
arcgis_cloud_platform=null
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ProcessMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'DirectoryCleaner'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'UlimitMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'LogFileLimiter'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ConfigSynchronizer'.
Tue Dec 19 09:40:07 GMT 2017:Loading plugin.. com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor
Tue Dec 19 09:40:07 GMT 2017:Props {PeriodicInterval=1, NodeAgentCustomPlugin=false}
Tue Dec 19 09:40:07 GMT 2017:Loaded plugin com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ServiceInstanceMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'PlatformServicesPluginObserver'.
Tue Dec 19 09:40:07 GMT 2017:Starting all plugins.z-service-gistest
StartRMID - arg: -J-Djava.security.policy=file:/C:/Program%20Files/ArcGIS/Server/framework/etc/rmid.policy
StartRMID - arg: -J-Dsun.rmi.activation.execPolicy=com.esri.arcgis.discovery.servicelib.impl.RMIDPolicyChecker
StartRMID - arg: "-J-Djava.class.path=C:\Program Files\ArcGIS\Server\framework\lib\server\arcgis-servicelib.jar;C:\Program Files\ArcGIS\Server\framework\lib\server\arcgis-common.jar"
StartRMID - arg: -J-Djava.rmi.server.useCodebaseOnly=true
StartRMID - arg: -port
StartRMID - arg: 1098
Constructing RMIDPolicyChecker: start
Constructing RMIDPolicyChecker: done
Using CATALINA_BASE: "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat"
Using CATALINA_HOME: "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat"
Using CATALINA_TMPDIR: "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\temp"
Using ARCGIS_JAVA_HOME: "C:\Program Files\ArcGIS\Server\framework/runtime/jre"
Using CLASSPATH: "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\bin\bootstrap.jar;C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\bin\tomcat-juli.jar"
Tue Dec 19 10:10:13 GMT 2017:Invoking afterStart() for all observers.
Tue Dec 19 10:10:16 GMT 2017:End NodeAgent.start().

Regards,

Apurv

0 Kudos
JohnAcosta
New Contributor

I have the same problem. Arcgis Server 10.2 for Linux Red Hat.

0 Kudos
RandallWilliams
Esri Regular Contributor

The log reads like somethings blocking access to RMI, which ArcGIS Server uses internally when publishing services. By default RMI runs on port 1098, and should increment up if that port is in use.

http://enterprise.arcgis.com/en/server/latest/install/linux/ports-used-by-arcgis-server.htm