SuSE 11 and ArcServer 10.2 wont start JMX error

6576
13
10-31-2013 11:08 AM
TimOlow
New Contributor
Hello all,

Im trying to do some performance benchmark testing against Windows and Linux versions of arcserver.  Im attempting to bring up a node running SuSE 11 and ArcServer.  I am running into issues after install, when trying to start the server using the builtin startserver.sh I see the following error messages in: $installdir/framework/etc/*.log

==> service.log <==
Thu Oct 31 18:03:35 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent
Start invoked
Thu Oct 31 18:29:47 CDT 2013:Start NodeAgent.start().
Thu Oct 31 18:29:47 CDT 2013:Loading all observers.
arcgis_cloud_platform=null
Thu Oct 31 18:29:48 CDT 2013:NodeAgent loaded observer 'Server upgrader'.
Thu Oct 31 18:29:48 CDT 2013:Regular startup of Server.
Thu Oct 31 18:29:48 CDT 2013:Invoking beforeStart() for all observers.
Thu Oct 31 18:29:48 CDT 2013:Starting RMI connector for NodeAgent.
Thu Oct 31 18:30:00 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent
Start invoked
Thu Oct 31 18:58:10 CDT 2013:Start NodeAgent.start().
Thu Oct 31 18:58:10 CDT 2013:Loading all observers.
arcgis_cloud_platform=null
Thu Oct 31 18:58:10 CDT 2013:NodeAgent loaded observer 'Server upgrader'.
Thu Oct 31 18:58:10 CDT 2013:Regular startup of Server.
Thu Oct 31 18:58:10 CDT 2013:Invoking beforeStart() for all observers.
Thu Oct 31 18:58:10 CDT 2013:Starting RMI connector for NodeAgent.
Thu Oct 31 18:58:23 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent

==> service_error.log <==
java.lang.RuntimeException: Failed to start JMX Server
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:44)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:670)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:276)
at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)
at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)
Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
at java.net.ServerSocket.<init>(ServerSocket.java:232)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at com.esri.arcgis.discovery.util.IPUtil.isPortAvailable(IPUtil.java:628)
at com.esri.arcgis.discovery.util.JMXServerController.c(JMXServerController.java:162)
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:35)
... 4 more
Exception in thread "main" com.esri.arcgis.discovery.nodeagent.NodeAgentException: Could not start RMI connector. An instance of NodeAgent may be running or the JMX-RMI port is in use by another process.
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:279)
at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)
at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)
Caused by: java.lang.RuntimeException: Failed to start JMX Server
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:44)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:670)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:276)
... 2 more
Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
at java.net.ServerSocket.<init>(ServerSocket.java:232)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at com.esri.arcgis.discovery.util.IPUtil.isPortAvailable(IPUtil.java:628)
at com.esri.arcgis.discovery.util.JMXServerController.c(JMXServerController.java:162)
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:35)
... 4 more

localadmin@bigsuse:/data/arcgis/server/tools/serverdiag> ./serverdiag
========================================================================
                 ArcGIS 10.2 for Server Diagnostic Tool
                           Hostname: bigsuse
========================================================================

DIAG000: Check for installation as root                       [PASSED]
DIAG001: Check for 64-bit architecture                        [PASSED]
DIAG002: Check OS version                                     [PASSED]
DIAG003: Check hostname for invalid characters                [PASSED]
DIAG004: Check installed packages                             [PASSED]
DIAG005: Check system limits                                  [PASSED]
DIAG006: Check OS patches                                     [N/A]
DIAG008: Check HTTP port                                      [PASSED]
DIAG009: Check HTTPS port                                     [PASSED]
DIAG010: Check Xvfb ports                                     [PASSED]
DIAG020: Check hostname IP address mismatches                 [PASSED]
DIAG026: Check processes for ArcGIS core services             [WARNING]

------------------------------------------------------------------------
There were 0 failure(s) and 1 warning(s) found:


WARNINGS:
------------------------------------------------------------------------
*** DIAG026: One or more core services was not started.  Start ArcGIS
for Server by running the startserver.sh script or restart ArcGIS for
Server if it is already started.



Any ideas why JMX is trying to capture a port outside of the normal 65535?
0 Kudos
13 Replies
JustinRodriguez
Occasional Contributor
Hello

I was wondering if your handles and procs were set:


To increase the soft and hard limits, you'll need to edit the /etc/security/limits.conf file with superuser access. For example, you can add four lines like this in the file:
<ArcGIS Server installation user> soft nofile 65535
<ArcGIS Server installation user> hard nofile 65535
<ArcGIS Server installation user> soft nproc 25059
<ArcGIS Server installation user> hard nproc 25059


Thanks-
Justin
0 Kudos
TimOlow
New Contributor
Justin,

Here is my limits.conf file "localadmin" is the account arcserver is running under:

bigsuse:~ # cat /etc/security/limits.conf  | grep localadmin
localadmin soft nofile 65535
localadmin hard nofile 65535
localadmin soft nproc 25059
localadmin hard nproc 25059


Tim
Hello

I was wondering if your handles and procs were set:


To increase the soft and hard limits, you'll need to edit the /etc/security/limits.conf file with superuser access. For example, you can add four lines like this in the file:
<ArcGIS Server installation user> soft nofile 65535
<ArcGIS Server installation user> hard nofile 65535
<ArcGIS Server installation user> soft nproc 25059
<ArcGIS Server installation user> hard nproc 25059


Thanks-
Justin
0 Kudos
JustinRodriguez
Occasional Contributor
Hello again,
hmm.. I still think this is a ulimit issue. I have seen some forums state that limits set in the .kshrc file prevent them from properly setting the ulimits in the limits.conf. Can you remove any ulimits in the .kshrc file and see if there is any difference. The forums I found were actually referring to Oracle settings, but I think the fix might apply in this case. Thank you very much-

Justin
0 Kudos
TimOlow
New Contributor
Justin,

I just checked right now, here is the current limits for the user running arcserver.

localadmin@bigsuse:~> ulimit -Hn
65535
localadmin@bigsuse:~> ulimit -Hu
25059
localadmin@bigsuse:~> ulimit -Su
25059
localadmin@bigsuse:~> ulimit -Sn
65535
localadmin@bigsuse:~>


Hello again,
hmm.. I still think this is a ulimit issue. I have seen some forums state that limits set in the .kshrc file prevent them from properly setting the ulimits in the limits.conf. Can you remove any ulimits in the .kshrc file and see if there is any difference. The forums I found were actually referring to Oracle settings, but I think the fix might apply in this case. Thank you very much-

Justin
0 Kudos
JustinRodriguez
Occasional Contributor
Thank you Tim,
Did you get a chance to check your .kshrc file? Thanks-

Justin

Justin,

I just checked right now, here is the current limits for the user running arcserver.

localadmin@bigsuse:~> ulimit -Hn
65535
localadmin@bigsuse:~> ulimit -Hu
25059
localadmin@bigsuse:~> ulimit -Su
25059
localadmin@bigsuse:~> ulimit -Sn
65535
localadmin@bigsuse:~>
0 Kudos
JustinRodriguez
Occasional Contributor
Tim,
Can you also give a netstat -ao (I think that is the unix command).

We need to ensure that nothing is listening on the ports listed in this web help:

http://resources.arcgis.com/en/help/main/10.1/index.html#//015400000537000000
0 Kudos
TimOlow
New Contributor
Justin,

Thanks for the reply, however the port that it is trying to bind to is 1 port above the supported standard. 

bigsuse:~ # netstat -ao
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       Timer
tcp        0      0 *:ssh                   *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:webyast               *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:smtp                  *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:mshvlm                *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:5801                  *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:5901                  *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:sunrpc                *:*                     LISTEN      off (0.00/0/0)
tcp        0    624 bigsuse.na.drilling:ssh 10.10.96.124:52900      ESTABLISHED on (0.20/0/0)
tcp        0      0 bigsuse.na.drilling:ssh 10.10.96.124:63362      ESTABLISHED keepalive (2323.11/0/0)
tcp        0      0 *:ssh                   *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:smtp                  *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:mshvlm                *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:sunrpc                *:*                     LISTEN      off (0.00/0/0)
udp        0      0 *:sunrpc                *:*                                 off (0.00/0/0)
udp        0      0 *:iclcnet_svinfo        *:*                                 off (0.00/0/0)
udp        0      0 *:sunrpc                *:*                                 off (0.00/0/0)
udp        0      0 *:xdmcp                 *:*                                 off (0.00/0/0)
udp        0      0 *:iclcnet_svinfo        *:*                                 off (0.00/0/0)

Tim


Tim,
Can you also give a netstat -ao (I think that is the unix command).

We need to ensure that nothing is listening on the ports listed in this web help:

http://resources.arcgis.com/en/help/main/10.1/index.html#//015400000537000000
0 Kudos
MarcosBusby
New Contributor
Did you ever get this resolved? I am seeing the identical issue on a RHEL 6.5 system.
0 Kudos
TimOlow
New Contributor
Did you ever get this resolved? I am seeing the identical issue on a RHEL 6.5 system.


No i did not, are you in a VM environment?


Tim
0 Kudos