SuSE 11 and ArcServer 10.2 wont start JMX error

TimOlow · ‎10-31-2013

Hello all,

Im trying to do some performance benchmark testing against Windows and Linux versions of arcserver. Im attempting to bring up a node running SuSE 11 and ArcServer. I am running into issues after install, when trying to start the server using the builtin startserver.sh I see the following error messages in: $installdir/framework/etc/*.log

==> service.log <==
Thu Oct 31 18:03:35 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent
Start invoked
Thu Oct 31 18:29:47 CDT 2013:Start NodeAgent.start().
Thu Oct 31 18:29:47 CDT 2013:Loading all observers.
arcgis_cloud_platform=null
Thu Oct 31 18:29:48 CDT 2013:NodeAgent loaded observer 'Server upgrader'.
Thu Oct 31 18:29:48 CDT 2013:Regular startup of Server.
Thu Oct 31 18:29:48 CDT 2013:Invoking beforeStart() for all observers.
Thu Oct 31 18:29:48 CDT 2013:Starting RMI connector for NodeAgent.
Thu Oct 31 18:30:00 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent
Start invoked
Thu Oct 31 18:58:10 CDT 2013:Start NodeAgent.start().
Thu Oct 31 18:58:10 CDT 2013:Loading all observers.
arcgis_cloud_platform=null
Thu Oct 31 18:58:10 CDT 2013:NodeAgent loaded observer 'Server upgrader'.
Thu Oct 31 18:58:10 CDT 2013:Regular startup of Server.
Thu Oct 31 18:58:10 CDT 2013:Invoking beforeStart() for all observers.
Thu Oct 31 18:58:10 CDT 2013:Starting RMI connector for NodeAgent.
Thu Oct 31 18:58:23 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent

==> service_error.log <==
java.lang.RuntimeException: Failed to start JMX Server
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:44)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:670)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:276)
at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)
at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)
Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
at java.net.ServerSocket.<init>(ServerSocket.java:232)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at com.esri.arcgis.discovery.util.IPUtil.isPortAvailable(IPUtil.java:628)
at com.esri.arcgis.discovery.util.JMXServerController.c(JMXServerController.java:162)
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:35)
... 4 more
Exception in thread "main" com.esri.arcgis.discovery.nodeagent.NodeAgentException: Could not start RMI connector. An instance of NodeAgent may be running or the JMX-RMI port is in use by another process.
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:279)
at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)
at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)
Caused by: java.lang.RuntimeException: Failed to start JMX Server
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:44)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:670)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:276)
... 2 more
Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
at java.net.ServerSocket.<init>(ServerSocket.java:232)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at com.esri.arcgis.discovery.util.IPUtil.isPortAvailable(IPUtil.java:628)
at com.esri.arcgis.discovery.util.JMXServerController.c(JMXServerController.java:162)
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:35)
... 4 more

localadmin@bigsuse:/data/arcgis/server/tools/serverdiag> ./serverdiag
========================================================================
                 ArcGIS 10.2 for Server Diagnostic Tool
                           Hostname: bigsuse
========================================================================

DIAG000: Check for installation as root                       [PASSED]
DIAG001: Check for 64-bit architecture                        [PASSED]
DIAG002: Check OS version                                     [PASSED]
DIAG003: Check hostname for invalid characters                [PASSED]
DIAG004: Check installed packages                             [PASSED]
DIAG005: Check system limits                                  [PASSED]
DIAG006: Check OS patches                                     [N/A]
DIAG008: Check HTTP port                                      [PASSED]
DIAG009: Check HTTPS port                                     [PASSED]
DIAG010: Check Xvfb ports                                     [PASSED]
DIAG020: Check hostname IP address mismatches                 [PASSED]
DIAG026: Check processes for ArcGIS core services             [WARNING]

------------------------------------------------------------------------
There were 0 failure(s) and 1 warning(s) found:

WARNINGS:
------------------------------------------------------------------------
*** DIAG026: One or more core services was not started. Start ArcGIS
for Server by running the startserver.sh script or restart ArcGIS for
Server if it is already started.

Any ideas why JMX is trying to capture a port outside of the normal 65535?

JustinRodriguez · ‎01-09-2014

No i did not, are you in a VM environment?

Tim

Hello Everyone,
So this error happens in other programs besides ours. It is a Java error stating that JMX tried to open basically on every port possible and couldn't. The way the error is written states that it went out of range (65536), but that is because the upper limit is 65535.

The real question here is why JMX couldn't open on any port? What we need to investigate is your network connection:

1. Do you have more than one NIC? If so, can you disable all NICs but the primary and try again?

2. Is DNS set up correctly? Does NSLookup return a value that is correct?

3. Do you have any conflicts in your etc/hosts file? A good way to test this would be to simply comment out anything other than 'localhost' entries. *literally anything that doesn't relate to the word 'localhost'. Also all 'localhost' entries should only point to loop back IP addresses, and nothing else.

4. Do you have iptables enabled? Can you flush them temporarily?

5. Does your Server have SELinux turned on? Can you put it in permissive mode temporarily?

These are just some examples that I can think about now. Can you guys think of anything else that might be causing this type of issue?

I would like to note that Redhat 6.5 isn't certified as of yet, and only SUSE SP1 I believe is certified at the moment as well. It could be a bug, so I am interested in hearing more about the issue. Thank you very much-

Justin Rodriguez

WK1 · ‎06-07-2014

I'm getting this error as well, but it is on Windows.

And to answer your first question, the machine has many NICs and disabling them is not an option.

Hence I conclude this is likely a bug with the JMX initialization code that is being used...

philippenn · ‎07-21-2015

Had same error on Windows, too. Turned out to be an updated Macafee firewall thing. Switched off firewall, all good.

Tue Jul 21 18:10:26 PDT 2015:Registering and starting log service.

Tue Jul 21 18:11:26 PDT 2015:ERROR: Failed to start the log service.

Tue Jul 21 18:11:26 PDT 2015:java.lang.RuntimeException: java.lang.Exception: Another administrative operation is curren

tly accessing the store. Please try again later.

at com.esri.arcgis.discovery.logging.impl.LogService.d(LogService.java:328)

at com.esri.arcgis.discovery.logging.impl.LogService.<init>(LogService.java:94)

at com.esri.arcgis.discovery.logging.impl.LogServiceLoader.createLogService(LogServiceLoader.java:17)

at com.esri.arcgis.discovery.admin.plugins.LogServicePlugin.a(LogServicePlugin.java:56)

at com.esri.arcgis.discovery.admin.plugins.LogServicePlugin.beforeStart(LogServicePlugin.java:40)

at com.esri.arcgis.discovery.nodeagent.impl.ObserverManager.beforeStart(ObserverManager.java:141)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.d(NodeAgent.java:290)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:158)

at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)

at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)

Caused by: java.lang.Exception: Another administrative operation is currently accessing the store. Please try again late

r.

at com.esri.arcgis.discovery.util.LockUtil.lockAndReadFileAsString(LockUtil.java:93)

at com.esri.arcgis.discovery.logging.impl.LogService.d(LogService.java:285)

... 9 more

Tue Jul 21 18:11:26 PDT 2015:Starting RMI connector for NodeAgent.

Tue Jul 21 18:11:27 PDT 2015:ERROR: Unable to start the RMI connector for NodeAgent

Tue Jul 21 18:11:27 PDT 2015:java.lang.RuntimeException: Failed to start JMX Server

at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:45)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:509)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:164)

at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)

at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)

Caused by: java.lang.NullPointerException

at com.esri.arcgis.discovery.util.a.a(a.java:81)

at com.esri.arcgis.discovery.util.a.<init>(a.java:39)

at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:41)

... 4 more

Exception in thread "main" com.esri.arcgis.discovery.nodeagent.NodeAgentException: Could not start RMI connector. An ins

tance of NodeAgent may be running or the JMX-RMI port is in use by another process.

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:167)

at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)

at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)

Caused by: java.lang.RuntimeException: Failed to start JMX Server

at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:45)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:509)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:164)

... 2 more

Caused by: java.lang.NullPointerException

at com.esri.arcgis.discovery.util.a.a(a.java:81)

at com.esri.arcgis.discovery.util.a.<init>(a.java:39)

at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:41)

... 4 more

Edgar_W_Iparraguirre · ‎12-04-2021

Old issue ... but just today came against on a VM running CentOS7 + AGS 10.9.1 ... it took me a while but I have found a solution (which counts in my case): on the VM manager generated new MAC-addresses (IPv4, IPvv6) ... and problem solved.

dmesg put me on the right track, as there were strings indicating a conflict with the MAC-addresses.

Edgar W. Iparraguirre