ArcGIS Server 10.5 server starting but services failing to start

Question

ArcGIS Enterprise 10.5 installed on OEL 7.2. The serverdiag seems to be OK with the warning of core services not starting. The service-error.log file indicates error in Java below. The service.log seems OK. But the problem is none of the services are starting. We just recognized that the ArcGIS server wasn't working probably for 2-3 days today, so we can not trace back to what the problem might be. I also changed the SERVER TMP in usr/init_user_param.sh but that didn't work also after a server stop and start. Any suggestions from the below logs of why the services might not be starting? Service-error.log file contents: Mar 30, 2017 2:50:23 PM ClientCommunicatorAdmin restart WARNING: Failed to restart: java.rmi.NoSuchObjectException: no such object in table Mar 30, 2017 2:50:23 PM ClientCommunicatorAdmin restart WARNING: Failed to restart: java.io.IOException: Failed to get a RMI stub: javax.naming.NameNotFoundException: jmxrmi java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:136) at com.esri.arcgis.discovery.admin.util.HttpUtil.postToURL(HttpUtil.java:405) at com.esri.arcgis.discovery.admin.util.HttpUtil.postPropsToURL(HttpUtil.java:329) at com.esri.arcgis.discovery.admin.system.handlers.RestConfigManager$a.call(RestConfigManager$a.java:710) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Service.log file contents (last 100 lines) JMX Closed WARNING: Could not connect to Log service. Writing log message to console. The machine '10.0.87.21' was not reachable while performing this operation. The server machine will be notified of this change when it comes back online. If ArcGIS Server is going to continue to be unavailable, remove the machine from the site. Start invoked Thu Mar 30 14:50:33 AZT 2017:Start NodeAgent.start(). Thu Mar 30 14:50:33 AZT 2017:Loading all observers. arcgis_cloud_platform=null arcgis_cloud_platform=null Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'LogServiceObserver'. Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'ServerUninstallObserver'. Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'ServerConfigObserver'. Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'. Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'. Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'Server upgrader'. Thu Mar 30 14:50:33 AZT 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'. Thu Mar 30 14:50:33 AZT 2017:Regular startup of Server. Thu Mar 30 14:50:33 AZT 2017:Starting RMI connector for NodeAgent. Thu Mar 30 14:50:34 AZT 2017:Invoking beforeStart() for all observers. Thu Mar 30 14:50:34 AZT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi17983:9 Thu Mar 30 14:50:34 AZT 2017:Registering and starting log service on TEKUISPARCGIS01.EMDKDEDRX.GOV.AZ. Thu Mar 30 14:50:34 AZT 2017:Registering NodeAgent as JMX bean. Thu Mar 30 14:50:34 AZT 2017:Loading all plugins. arcgis_cloud_platform=null Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ProcessMonitor'. Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'DirectoryCleaner'. Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'UlimitMonitor'. Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'LogFileLimiter'. Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ConfigSynchronizer'. Thu Mar 30 14:50:34 AZT 2017:Loading plugin.. com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor Thu Mar 30 14:50:34 AZT 2017:Props {PeriodicInterval=1, NodeAgentCustomPlugin=false} Thu Mar 30 14:50:34 AZT 2017:Loaded plugin com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'ServiceInstanceMonitor'. Thu Mar 30 14:50:34 AZT 2017:NodeAgent loaded plugin 'PlatformServicesPluginObserver'. Thu Mar 30 14:50:34 AZT 2017:Starting all plugins.arcgis StartRMID - arg: -J-Djava.security.policy=file:////home/arcgis/server/framework/etc/rmid.policy StartRMID - arg: -J-Dsun.rmi.activation.execPolicy=com.esri.arcgis.discovery.servicelib.impl.RMIDPolicyChecker StartRMID - arg: -J-Djava.class.path=/home/arcgis/server/framework/lib/server/arcgis-servicelib.jar:/home/arcgis/server/framework/lib/server/arcgis-common.jar StartRMID - arg: -J-Djava.rmi.server.useCodebaseOnly=true StartRMID - arg: -port StartRMID - arg: 1098 Using GERONIMO_HOME: /home/arcgis/server/geronimo Using GERONIMO_TMPDIR: var/temp Using ARCGIS_JAVA_HOME: /home/arcgis/server//framework/runtime/jre Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 Booting Geronimo Kernel (in Java 1.8.0_101)... Starting Geronimo Application Server v2.2.2-SNAPSHOT [****************************************] 100% 2s Startup complete Listening on Ports: 4000 0.0.0.0 OpenEJB Daemon 4001 0.0.0.0 JMX Remoting Connector 4002 0.0.0.0 RMI Naming 4003 0.0.0.0 Derby Connector Started Application Modules: JAR: com.esri.arcgis.services/GeronimoSingleton/1.0/car JAR: com.esri.arcgis.services/ServiceCatalog/1.0/car JAR: com.esri.arcgis.services/System.CachingToolsEx.GPServer/1.0/car JAR: com.esri.arcgis.services/System.DynamicMappingHost.MapServer/1.0/car JAR: com.esri.arcgis.services/System.PublishingToolsEx.GPServer/1.0/car JAR: org.apache.geronimo.configs/mejb/2.2.2-SNAPSHOT/car RAR: org.apache.geronimo.configs/system-database/2.2.2-SNAPSHOT/car Geronimo Application Server started system.reg is not a valid registry file userdef.reg:4: Mismatched architecture '#arch=win64' userdef.reg is not a valid registry file user.reg:4: Mismatched architecture '#arch=win64' user.reg is not a valid registry file wine: '/home/arcgis/server/framework/runtime/.wine' is a 32-bit installation, it cannot support 64-bit applications.

RandallWilliams · Answer

The log reads like somethings blocking access to RMI, which ArcGIS Server uses internally when publishing services. By default RMI runs on port 1098, and should increment up if that port is in use.

http://enterprise.arcgis.com/en/server/latest/install/linux/ports-used-by-arcgis-server.htm

JohnAcosta · Answer

I have the same problem. Arcgis Server 10.2 for Linux Red Hat.

ApurvDanke · Answer

Hi All,

I am getting a similar error in the service.log file(as given below). The ArcGIS Server version is 10.4.1 for Windows. The operating system is Windows Server 2012 R2. Due to this we're getting error while making a connection in ArcCatalog to administer ArcGIS Server. That error message is ->

Tue Dec 19 06:55:00 GMT 2017:End NodeAgent.stop(true).
Start invoked
Tue Dec 19 09:40:01 GMT 2017:Start NodeAgent.start().
Tue Dec 19 09:40:01 GMT 2017:Loading all observers.
arcgis_cloud_platform=null
arcgis_cloud_platform=null
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'LogServiceObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'ServerUninstallObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'ServerConfigObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'Server upgrader'.
Tue Dec 19 09:40:03 GMT 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'.
Tue Dec 19 09:40:03 GMT 2017:Regular startup of Server.
Tue Dec 19 09:40:03 GMT 2017:Starting RMI connector for NodeAgent.
Tue Dec 19 09:40:04 GMT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
 java.net.ConnectException: Connection refused: connect]20948:10
Tue Dec 19 09:40:04 GMT 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi20948:12
Tue Dec 19 09:40:04 GMT 2017:Invoking beforeStart() for all observers.
Tue Dec 19 09:40:05 GMT 2017:Registering and starting log service on <arcgis_server_hostname>.
Tue Dec 19 09:40:07 GMT 2017:Registering NodeAgent as JMX bean.
Tue Dec 19 09:40:07 GMT 2017:Loading all plugins.
arcgis_cloud_platform=null
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ProcessMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'DirectoryCleaner'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'UlimitMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'LogFileLimiter'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ConfigSynchronizer'.
Tue Dec 19 09:40:07 GMT 2017:Loading plugin.. com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor
Tue Dec 19 09:40:07 GMT 2017:Props {PeriodicInterval=1, NodeAgentCustomPlugin=false}
Tue Dec 19 09:40:07 GMT 2017:Loaded plugin com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'ServiceInstanceMonitor'.
Tue Dec 19 09:40:07 GMT 2017:NodeAgent loaded plugin 'PlatformServicesPluginObserver'.
Tue Dec 19 09:40:07 GMT 2017:Starting all plugins.z-service-gistest
StartRMID - arg: -J-Djava.security.policy=file:/C:/Program%20Files/ArcGIS/Server/framework/etc/rmid.policy
StartRMID - arg: -J-Dsun.rmi.activation.execPolicy=com.esri.arcgis.discovery.servicelib.impl.RMIDPolicyChecker
StartRMID - arg: "-J-Djava.class.path=C:\Program Files\ArcGIS\Server\framework\lib\server\arcgis-servicelib.jar;C:\Program Files\ArcGIS\Server\framework\lib\server\arcgis-common.jar"
StartRMID - arg: -J-Djava.rmi.server.useCodebaseOnly=true
StartRMID - arg: -port
StartRMID - arg: 1098
Constructing RMIDPolicyChecker: start
Constructing RMIDPolicyChecker: done
Using CATALINA_BASE: "C:\Program Files\ArcGIS\Server\frameworkuntime	omcat"
Using CATALINA_HOME: "C:\Program Files\ArcGIS\Server\frameworkuntime	omcat"
Using CATALINA_TMPDIR: "C:\Program Files\ArcGIS\Server\frameworkuntime	omcat	emp"
Using ARCGIS_JAVA_HOME: "C:\Program Files\ArcGIS\Server\framework/runtime/jre"
Using CLASSPATH: "C:\Program Files\ArcGIS\Server\frameworkuntime	omcat\bin\bootstrap.jar;C:\Program Files\ArcGIS\Server\frameworkuntime	omcat\bin	omcat-juli.jar"
Tue Dec 19 10:10:13 GMT 2017:Invoking afterStart() for all observers.
Tue Dec 19 10:10:16 GMT 2017:End NodeAgent.start().

Regards,

Apurv