The following message is appearing in my log file. Not sure where to begin troubleshooting. This is a new server, clean install of Windows Server 2012 r2. ArcGIS Server 10.3 and GeoEvent processor installed. Any advice on where to look for the cause of this error would be greatly apprecitated! Thanks!
2015-05-21 13:15:17,043 | ERROR | Thread-45 | ConnectionHandler | .lyra.internal.ConnectionHandler 125 | 243 - com.esri.ges.framework.messaging.jms-messaging - 10.3.0 | Failed to create connection cxn-160971 java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)[:1.7.0_65]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)[:1.7.0_65]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)[:1.7.0_65]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)[:1.7.0_65]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)[:1.7.0_65]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)[:1.7.0_65]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)[:1.7.0_65]
at java.net.Socket.connect(Socket.java:579)[:1.7.0_65]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)[:1.7.0_65]
at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:32)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:588)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:233)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:227)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at net.jodah.lyra.internal.RetryableResource.callWithRetries(RetryableResource.java:50)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:227)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:115)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at net.jodah.lyra.Connections.create(Connections.java:68)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at com.esri.ges.messaging.jms.AmqpMessagingFactory$ConnectionEstablisher.run(AmqpMessagingFactory.java:669)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
Sharon -
The log messages you provide above look to me like the Rabbit MQ message broker service (provided by ArcGIS Server) has failed or is not running.
There are some trouble shooting steps I can share with you that are found in the product tutorial for clustering. Technically even if you are configuring a Server site with only a single server machine, you are still running a cluster - just with only one machine.
Before we go into that, however, I'd like to get your feedback on the questions above.
Thanks -
RJ
Hi, and thanks for the reply. Yes, I can launch GeoEvent manager web application. I have created input connector that is receiving data and incrementing but the service in/out counts are remaining at zero. That's another issue I've been trying to track down, but based on your responses it sounds like it is related to this error message. I will look at the information in your other log entry and try those steps.
If you suspect that ArcGIS Server has failed to correctly configure the RabbitMQ platform service:
The only file you should find here should be a file named site.rlock
If you discover any files in …\config-store\locks whose names match the pattern machine-<MACHINE_NAME>.wlock or machine-<MACHINE_NAME>.wlock.info:
The illustration above only shows a couple of the processes frequently found to still be running when the RabbitMQ platform service failed to initialize correctly and the ArcGIS Server Windows Service has been shut down.
With AGS stopped, there should be no processes running whose command line path indicates they were launched from either:
- C:\Program Files\ArcGIS\Server
- C:\PROGRA~1\ArcGIS\Server
Once you are able to launch and log-in to the ArcGIS Server Manager web application, go ahead and restart the GeoEvent Windows Service.
I've deleted the .wlock and .wlock.info files that were in my \config-store\locks folder. Restarted ArcGIS Server service and GeoEvent service. I'm still getting the error in my log file. How can I confirm that RabbitMQ is running?
Sharon -
I've created a branch for this discussion to place information on a diagnostic utility you can use to check the status of the RabbitMQ platform service in a separate thread.
Please check-out my post 'Diagnostic utility for checking the status of the RabbitMQ platform service'.
- RJ
Sharon,
In order for this trick to work, you must stop the Windows services for GeoEvent and ArcGIS Server first. Once they're both stopped, then you go and check the config folder for the *.wlock or *.wlock.info files and kill any processes that may still be running that show as being launch from C:\Program Files\ArcGIS\Server or C:\PROGRA~1\ArcGIS\Server. Once you've done both of those with the services not running, then restart ArcGIS Server followed by GeoEvent.
This sequence usually rescues RabbitMQ for me, but the order is important.
Can you please confirm you tried doing the steps in the order mentioned above?
-Javier
Javier,
Yes, I tried the steps in the order listed. Just tried them a second time, again in the order listed, and am getting the same result. It appears that RabbitMQ is not running.
Javier, tried the steps again yesterday. The third time it worked.
Good to hear. Sorry it took so many tries.
-Javier