nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

SuSE 11 and ArcServer 10.2 wont start JMX error

TimOlow

Hello all,

Im trying to do some performance benchmark testing against Windows and Linux versions of arcserver. Im attempting to bring up a node running SuSE 11 and ArcServer. I am running into issues after install, when trying to start the server using the builtin startserver.sh I see the following error messages in: $installdir/framework/etc/*.log

==> service.log <==
Thu Oct 31 18:03:35 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent
Start invoked
Thu Oct 31 18:29:47 CDT 2013:Start NodeAgent.start().
Thu Oct 31 18:29:47 CDT 2013:Loading all observers.
arcgis_cloud_platform=null
Thu Oct 31 18:29:48 CDT 2013:NodeAgent loaded observer 'Server upgrader'.
Thu Oct 31 18:29:48 CDT 2013:Regular startup of Server.
Thu Oct 31 18:29:48 CDT 2013:Invoking beforeStart() for all observers.
Thu Oct 31 18:29:48 CDT 2013:Starting RMI connector for NodeAgent.
Thu Oct 31 18:30:00 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent
Start invoked
Thu Oct 31 18:58:10 CDT 2013:Start NodeAgent.start().
Thu Oct 31 18:58:10 CDT 2013:Loading all observers.
arcgis_cloud_platform=null
Thu Oct 31 18:58:10 CDT 2013:NodeAgent loaded observer 'Server upgrader'.
Thu Oct 31 18:58:10 CDT 2013:Regular startup of Server.
Thu Oct 31 18:58:10 CDT 2013:Invoking beforeStart() for all observers.
Thu Oct 31 18:58:10 CDT 2013:Starting RMI connector for NodeAgent.
Thu Oct 31 18:58:23 CDT 2013:ERROR: Unable to start the RMI connector for NodeAgent

==> service_error.log <==
java.lang.RuntimeException: Failed to start JMX Server
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:44)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:670)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:276)
at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)
at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)
Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
at java.net.ServerSocket.<init>(ServerSocket.java:232)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at com.esri.arcgis.discovery.util.IPUtil.isPortAvailable(IPUtil.java:628)
at com.esri.arcgis.discovery.util.JMXServerController.c(JMXServerController.java:162)
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:35)
... 4 more
Exception in thread "main" com.esri.arcgis.discovery.nodeagent.NodeAgentException: Could not start RMI connector. An instance of NodeAgent may be running or the JMX-RMI port is in use by another process.
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:279)
at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)
at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)
Caused by: java.lang.RuntimeException: Failed to start JMX Server
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:44)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:670)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:276)
... 2 more
Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
at java.net.ServerSocket.<init>(ServerSocket.java:232)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at com.esri.arcgis.discovery.util.IPUtil.isPortAvailable(IPUtil.java:628)
at com.esri.arcgis.discovery.util.JMXServerController.c(JMXServerController.java:162)
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:35)
... 4 more

localadmin@bigsuse:/data/arcgis/server/tools/serverdiag> ./serverdiag
========================================================================
                 ArcGIS 10.2 for Server Diagnostic Tool
                           Hostname: bigsuse
========================================================================

DIAG000: Check for installation as root                       [PASSED]
DIAG001: Check for 64-bit architecture                        [PASSED]
DIAG002: Check OS version                                     [PASSED]
DIAG003: Check hostname for invalid characters                [PASSED]
DIAG004: Check installed packages                             [PASSED]
DIAG005: Check system limits                                  [PASSED]
DIAG006: Check OS patches                                     [N/A]
DIAG008: Check HTTP port                                      [PASSED]
DIAG009: Check HTTPS port                                     [PASSED]
DIAG010: Check Xvfb ports                                     [PASSED]
DIAG020: Check hostname IP address mismatches                 [PASSED]
DIAG026: Check processes for ArcGIS core services             [WARNING]

------------------------------------------------------------------------
There were 0 failure(s) and 1 warning(s) found:

WARNINGS:
------------------------------------------------------------------------
*** DIAG026: One or more core services was not started. Start ArcGIS
for Server by running the startserver.sh script or restart ArcGIS for
Server if it is already started.

Any ideas why JMX is trying to capture a port outside of the normal 65535?

Trouver plus de publications taguées avec

Accepted answers

All comments

JustinRodriguez

Hello

I was wondering if your handles and procs were set:

To increase the soft and hard limits, you'll need to edit the /etc/security/limits.conf file with superuser access. For example, you can add four lines like this in the file:
<ArcGIS Server installation user> soft nofile 65535
<ArcGIS Server installation user> hard nofile 65535
<ArcGIS Server installation user> soft nproc 25059
<ArcGIS Server installation user> hard nproc 25059

Thanks-
Justin

TimOlow

Justin,

Here is my limits.conf file "localadmin" is the account arcserver is running under:

bigsuse:~ # cat /etc/security/limits.conf | grep localadmin
localadmin soft nofile 65535
localadmin hard nofile 65535
localadmin soft nproc 25059
localadmin hard nproc 25059

Tim

Hello

I was wondering if your handles and procs were set:

To increase the soft and hard limits, you'll need to edit the /etc/security/limits.conf file with superuser access. For example, you can add four lines like this in the file:
<ArcGIS Server installation user> soft nofile 65535
<ArcGIS Server installation user> hard nofile 65535
<ArcGIS Server installation user> soft nproc 25059
<ArcGIS Server installation user> hard nproc 25059

Thanks-
Justin

JustinRodriguez

Hello again,
hmm.. I still think this is a ulimit issue. I have seen some forums state that limits set in the .kshrc file prevent them from properly setting the ulimits in the limits.conf. Can you remove any ulimits in the .kshrc file and see if there is any difference. The forums I found were actually referring to Oracle settings, but I think the fix might apply in this case. Thank you very much-

Justin

TimOlow

Justin,

I just checked right now, here is the current limits for the user running arcserver.

localadmin@bigsuse:~> ulimit -Hn
65535
localadmin@bigsuse:~> ulimit -Hu
25059
localadmin@bigsuse:~> ulimit -Su
25059
localadmin@bigsuse:~> ulimit -Sn
65535
localadmin@bigsuse:~>

Hello again,
hmm.. I still think this is a ulimit issue. I have seen some forums state that limits set in the .kshrc file prevent them from properly setting the ulimits in the limits.conf. Can you remove any ulimits in the .kshrc file and see if there is any difference. The forums I found were actually referring to Oracle settings, but I think the fix might apply in this case. Thank you very much-

Justin

JustinRodriguez

Thank you Tim,
Did you get a chance to check your .kshrc file? Thanks-

Justin

Justin,

I just checked right now, here is the current limits for the user running arcserver.

localadmin@bigsuse:~> ulimit -Hn
65535
localadmin@bigsuse:~> ulimit -Hu
25059
localadmin@bigsuse:~> ulimit -Su
25059
localadmin@bigsuse:~> ulimit -Sn
65535
localadmin@bigsuse:~>

JustinRodriguez

Tim,
Can you also give a netstat -ao (I think that is the unix command).

We need to ensure that nothing is listening on the ports listed in this web help:

http://resources.arcgis.com/en/help/main/10.1/index.html#//015400000537000000

TimOlow

Justin,

Thanks for the reply, however the port that it is trying to bind to is 1 port above the supported standard.

bigsuse:~ # netstat -ao
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       Timer
tcp        0      0 *:ssh                   *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:webyast               *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:smtp                  *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:mshvlm                *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:5801                  *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:5901                  *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:sunrpc                *:*                     LISTEN      off (0.00/0/0)
tcp        0    624 bigsuse.na.drilling:ssh 10.10.96.124:52900      ESTABLISHED on (0.20/0/0)
tcp        0      0 bigsuse.na.drilling:ssh 10.10.96.124:63362      ESTABLISHED keepalive (2323.11/0/0)
tcp        0      0 *:ssh                   *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:smtp                  *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:mshvlm                *:*                     LISTEN      off (0.00/0/0)
tcp        0      0 *:sunrpc                *:*                     LISTEN      off (0.00/0/0)
udp        0      0 *:sunrpc                *:*                                 off (0.00/0/0)
udp        0      0 *:iclcnet_svinfo        *:*                                 off (0.00/0/0)
udp        0      0 *:sunrpc                *:*                                 off (0.00/0/0)
udp        0      0 *:xdmcp                 *:*                                 off (0.00/0/0)
udp        0      0 *:iclcnet_svinfo        *:*                                 off (0.00/0/0)

Tim

Tim,
Can you also give a netstat -ao (I think that is the unix command).

We need to ensure that nothing is listening on the ports listed in this web help:

http://resources.arcgis.com/en/help/main/10.1/index.html#//015400000537000000

MarcosBusby

Did you ever get this resolved? I am seeing the identical issue on a RHEL 6.5 system.

TimOlow

Did you ever get this resolved? I am seeing the identical issue on a RHEL 6.5 system.

No i did not, are you in a VM environment?

Tim

JustinRodriguez

No i did not, are you in a VM environment?

Tim

Hello Everyone,
So this error happens in other programs besides ours. It is a Java error stating that JMX tried to open basically on every port possible and couldn't. The way the error is written states that it went out of range (65536), but that is because the upper limit is 65535.

The real question here is why JMX couldn't open on any port? What we need to investigate is your network connection:

1. Do you have more than one NIC? If so, can you disable all NICs but the primary and try again?

2. Is DNS set up correctly? Does NSLookup return a value that is correct?

3. Do you have any conflicts in your etc/hosts file? A good way to test this would be to simply comment out anything other than 'localhost' entries. *literally anything that doesn't relate to the word 'localhost'. Also all 'localhost' entries should only point to loop back IP addresses, and nothing else.

4. Do you have iptables enabled? Can you flush them temporarily?

5. Does your Server have SELinux turned on? Can you put it in permissive mode temporarily?

These are just some examples that I can think about now. Can you guys think of anything else that might be causing this type of issue?

I would like to note that Redhat 6.5 isn't certified as of yet, and only SUSE SP1 I believe is certified at the moment as well. It could be a bug, so I am interested in hearing more about the issue. Thank you very much-

Justin Rodriguez

WK1

I'm getting this error as well, but it is on Windows.

And to answer your first question, the machine has many NICs and disabling them is not an option.

Hence I conclude this is likely a bug with the JMX initialization code that is being used...

philippenn

Had same error on Windows, too. Turned out to be an updated Macafee firewall thing. Switched off firewall, all good.

Tue Jul 21 18:10:26 PDT 2015:Registering and starting log service.

Tue Jul 21 18:11:26 PDT 2015:ERROR: Failed to start the log service.

Tue Jul 21 18:11:26 PDT 2015:java.lang.RuntimeException: java.lang.Exception: Another administrative operation is curren

tly accessing the store. Please try again later.

at com.esri.arcgis.discovery.logging.impl.LogService.d(LogService.java:328)

at com.esri.arcgis.discovery.logging.impl.LogService.<init>(LogService.java:94)

at com.esri.arcgis.discovery.logging.impl.LogServiceLoader.createLogService(LogServiceLoader.java:17)

at com.esri.arcgis.discovery.admin.plugins.LogServicePlugin.a(LogServicePlugin.java:56)

at com.esri.arcgis.discovery.admin.plugins.LogServicePlugin.beforeStart(LogServicePlugin.java:40)

at com.esri.arcgis.discovery.nodeagent.impl.ObserverManager.beforeStart(ObserverManager.java:141)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.d(NodeAgent.java:290)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:158)

at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)

at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)

Caused by: java.lang.Exception: Another administrative operation is currently accessing the store. Please try again late

at com.esri.arcgis.discovery.util.LockUtil.lockAndReadFileAsString(LockUtil.java:93)

at com.esri.arcgis.discovery.logging.impl.LogService.d(LogService.java:285)

... 9 more

Tue Jul 21 18:11:26 PDT 2015:Starting RMI connector for NodeAgent.

Tue Jul 21 18:11:27 PDT 2015:ERROR: Unable to start the RMI connector for NodeAgent

Tue Jul 21 18:11:27 PDT 2015:java.lang.RuntimeException: Failed to start JMX Server

at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:45)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:509)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:164)

at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)

at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)

Caused by: java.lang.NullPointerException

at com.esri.arcgis.discovery.util.a.a(a.java:81)

at com.esri.arcgis.discovery.util.a.<init>(a.java:39)

at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:41)

... 4 more

Exception in thread "main" com.esri.arcgis.discovery.nodeagent.NodeAgentException: Could not start RMI connector. An ins

tance of NodeAgent may be running or the JMX-RMI port is in use by another process.

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:167)

at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:51)

at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:296)

Caused by: java.lang.RuntimeException: Failed to start JMX Server

at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:45)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:509)

at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:164)

... 2 more

Caused by: java.lang.NullPointerException

at com.esri.arcgis.discovery.util.a.a(a.java:81)

at com.esri.arcgis.discovery.util.a.<init>(a.java:39)

at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:41)

... 4 more

EdgarIparraguirre

Old issue ... but just today came against on a VM running CentOS7 + AGS 10.9.1 ... it took me a while but I have found a solution (which counts in my case): on the VM manager generated new MAC-addresses (IPv4, IPvv6) ... and problem solved.

dmesg put me on the right track, as there were strings indicating a conflict with the MAC-addresses.

Edgar W. Iparraguirre

Rejoindre la conversation

Les membres connectés peuvent publier, suivre les mises à jour, et plus encore. Nouveau ici ? Inscrivez-vous gratuitement.

Liens rapides

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.