topic ArcGIS Server 10.5 server starting but services failing to start in ArcGIS Enterprise Questions

ArcGIS Server 10.5 server starting but services failing to start

YucelTepekoy1 — Thu, 30 Mar 2017 10:59:53 GMT

ArcGIS Enterprise 10.5 installed on OEL 7.2. The serverdiag seems to be OK with the warning of core services not starting. The service-error.log file indicates error in Java below. The service.log seems OK. But the problem is none of the services are starting. We just recognized that the ArcGIS server wasn't working probably for 2-3 days today, so we can not trace back to what the problem might be. I also changed the SERVER TMP in usr/init_user_param.sh but that didn't work also after a server stop and start.

Any suggestions from the below logs of why the services might not be starting?

Service-error.log file contents:

Mar 30, 2017 2:50:23 PM ClientCommunicatorAdmin restart

WARNING: Failed to restart: java.rmi.NoSuchObjectException: no such object in table

Mar 30, 2017 2:50:23 PM ClientCommunicatorAdmin restart

WARNING: Failed to restart: java.io.IOException: Failed to get a RMI stub: javax.naming.NameNotFoundException: jmxrmi

java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)

at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)

at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)

at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)

at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)

at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:136)

at com.esri.arcgis.discovery.admin.util.HttpUtil.postToURL(HttpUtil.java:405)

at com.esri.arcgis.discovery.admin.util.HttpUtil.postPropsToURL(HttpUtil.java:329)

at com.esri.arcgis.discovery.admin.system.handlers.RestConfigManager$a.call(RestConfigManager$a.java:710)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Service.log file contents (last 100 lines)

JMX Closed

WARNING: Could not connect to Log service. Writing log message to console.

<Msg time="2017-03-30T14:50:23,943" type="WARNING" code="6740" source="Admin" process="14311" thread="1" methodName="" machine="" user="" elapsed="">The machine '10.0.87.21' was not reachable while performing this operation. The server machine will be notified of this change when it comes back online. If ArcGIS Server is going to continue to be unavailable, remove the machine from the site.</Msg>

Start invoked

Thu Mar 30 14:50:33 AZT 2017:Start NodeAgent.start().

Thu Mar 30 14:50:33 AZT 2017:Loading all observers.

arcgis_cloud_platform=null

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'LogServiceObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'ServerUninstallObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'ServerConfigObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'Server upgrader'.

Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'.

Thu Mar 30 14:50:33 AZT 2017:Regular startup of Server.

Thu Mar 30 14:50:33 AZT 2017:Starting RMI connector for NodeAgent.

Thu Mar 30 14:50:34 AZT 2017:Invoking beforeStart() for all observers.

Thu Mar 30 14:50:34 AZT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi17983:9

Thu Mar 30 14:50:34 AZT 2017:Registering and starting log service on TEKUISPARCGIS01.EMDKDEDRX.GOV.AZ.

Thu Mar 30 14:50:34 AZT 2017:Registering NodeAgent as JMX bean.

Thu Mar 30 14:50:34 AZT 2017:Loading all plugins.

arcgis_cloud_platform=null

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ProcessMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'DirectoryCleaner'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'UlimitMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'LogFileLimiter'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ConfigSynchronizer'.

Thu Mar 30 14:50:34 AZT 2017:Loading plugin.. com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor

Thu Mar 30 14:50:34 AZT 2017:Props {PeriodicInterval=1, NodeAgentCustomPlugin=false}

Thu Mar 30 14:50:34 AZT 2017:Loaded plugin com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ServiceInstanceMonitor'.

Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'PlatformServicesPluginObserver'.

Thu Mar 30 14:50:34 AZT 2017:Starting all plugins.arcgis

StartRMID - arg: -J-Djava.security.policy=file:////home/arcgis/server/framework/etc/rmid.policy

StartRMID - arg: -J-Dsun.rmi.activation.execPolicy=com.esri.arcgis.discovery.servicelib.impl.RMIDPolicyChecker

StartRMID - arg: -J-Djava.class.path=/home/arcgis/server/framework/lib/server/arcgis-servicelib.jar:/home/arcgis/server/framework/lib/server/arcgis-common.jar

StartRMID - arg: -J-Djava.rmi.server.useCodebaseOnly=true

StartRMID - arg: -port

StartRMID - arg: 1098

Using GERONIMO_HOME: /home/arcgis/server/geronimo

Using GERONIMO_TMPDIR: var/temp

Using ARCGIS_JAVA_HOME: /home/arcgis/server//framework/runtime/jre

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

Booting Geronimo Kernel (in Java 1.8.0_101)...

Starting Geronimo Application Server v2.2.2-SNAPSHOT

[****************************************] 100% 2s Startup complete

Listening on Ports:

4000 0.0.0.0 OpenEJB Daemon

4001 0.0.0.0 JMX Remoting Connector

4002 0.0.0.0 RMI Naming

4003 0.0.0.0 Derby Connector

Started Application Modules:

JAR: com.esri.arcgis.services/GeronimoSingleton/1.0/car

JAR: com.esri.arcgis.services/ServiceCatalog/1.0/car

JAR: com.esri.arcgis.services/System.CachingToolsEx.GPServer/1.0/car

JAR: com.esri.arcgis.services/System.DynamicMappingHost.MapServer/1.0/car

JAR: com.esri.arcgis.services/System.PublishingToolsEx.GPServer/1.0/car

JAR: org.apache.geronimo.configs/mejb/2.2.2-SNAPSHOT/car

RAR: org.apache.geronimo.configs/system-database/2.2.2-SNAPSHOT/car

Geronimo Application Server started

system.reg is not a valid registry file

userdef.reg:4: Mismatched architecture '#arch=win64'

userdef.reg is not a valid registry file

user.reg:4: Mismatched architecture '#arch=win64'

user.reg is not a valid registry file

wine: '/home/arcgis/server/framework/runtime/.wine' is a 32-bit installation, it cannot support 64-bit applications.

Re: ArcGIS Server 10.5 server starting but services failing to start

ApurvDanke — Tue, 19 Dec 2017 12:28:30 GMT

Hi All,

I am getting a similar error in the service.log file(as given below). The ArcGIS Server version is 10.4.1 for Windows. The operating system is Windows Server 2012 R2. Due to this we're getting error while making a connection in ArcCatalog to administer ArcGIS Server. That error message is ->

Tue Dec 19 06:55:00 GMT 2017:End NodeAgent.stop(true).
Start invoked
Tue Dec 19 09:40:01 GMT 2017:Start NodeAgent.start().
Tue Dec 19 09:40:01 GMT 2017:Loading all observers.
arcgis_cloud_platform=null
arcgis_cloud_platform=null
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'LogServiceObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'ServerUninstallObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'ServerConfigObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'Server upgrader'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'.
Tue Dec 19 09:40:03 GMT 2017:Regular startup of Server.
Tue Dec 19 09:40:03 GMT 2017:Starting RMI connector for NodeAgent.
Tue Dec 19 09:40:04 GMT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]20948:10
Tue Dec 19 09:40:04 GMT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi20948:12
Tue Dec 19 09:40:04 GMT 2017:Invoking beforeStart() for all observers.
Tue Dec 19 09:40:05 GMT 2017:Registering and starting log service on <arcgis_server_hostname>.
Tue Dec 19 09:40:07 GMT 2017:Registering NodeAgent as JMX bean.
Tue Dec 19 09:40:07 GMT 2017:Loading all plugins.
arcgis_cloud_platform=null
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ProcessMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'DirectoryCleaner'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'UlimitMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'LogFileLimiter'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ConfigSynchronizer'.
Tue Dec 19 09:40:07 GMT 2017:Loading plugin.. com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor
Tue Dec 19 09:40:07 GMT 2017:Props {PeriodicInterval=1, NodeAgentCustomPlugin=false}
Tue Dec 19 09:40:07 GMT 2017:Loaded plugin com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ServiceInstanceMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'PlatformServicesPluginObserver'.
Tue Dec 19 09:40:07 GMT 2017:Starting all plugins.z-service-gistest
StartRMID - arg: -J-Djava.security.policy=file:/C:/Program%20Files/ArcGIS/Server/framework/etc/rmid.policy
StartRMID - arg: -J-Dsun.rmi.activation.execPolicy=com.esri.arcgis.discovery.servicelib.impl.RMIDPolicyChecker
StartRMID - arg: "-J-Djava.class.path=C:\Program Files\ArcGIS\Server\framework\lib\server\arcgis-servicelib.jar;C:\Program Files\ArcGIS\Server\framework\lib\server\arcgis-common.jar"
StartRMID - arg: -J-Djava.rmi.server.useCodebaseOnly=true
StartRMID - arg: -port
StartRMID - arg: 1098
Constructing RMIDPolicyChecker: start
Constructing RMIDPolicyChecker: done
Using CATALINA_BASE: "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat"
Using CATALINA_HOME: "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat"
Using CATALINA_TMPDIR: "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\temp"
Using ARCGIS_JAVA_HOME: "C:\Program Files\ArcGIS\Server\framework/runtime/jre"
Using CLASSPATH: "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\bin\bootstrap.jar;C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\bin\tomcat-juli.jar"
Tue Dec 19 10:10:13 GMT 2017:Invoking afterStart() for all observers.
Tue Dec 19 10:10:16 GMT 2017:End NodeAgent.start().

Regards,

Apurv

Re: ArcGIS Server 10.5 server starting but services failing to start

JohnAcosta — Wed, 10 Jan 2018 19:38:43 GMT

I have the same problem. Arcgis Server 10.2 for Linux Red Hat.

Re: ArcGIS Server 10.5 server starting but services failing to start

RandallWilliams — Thu, 11 Jan 2018 15:55:54 GMT

The log reads like somethings blocking access to RMI, which ArcGIS Server uses internally when publishing services. By default RMI runs on port 1098, and should increment up if that port is in use.

http://enterprise.arcgis.com/en/server/latest/install/linux/ports-used-by-arcgis-server.htm