Error log entry:

7360
12
05-21-2015 01:11 PM
SharonMeier
New Contributor III

The following message is appearing in my log file.  Not sure where to begin troubleshooting.  This is a new server, clean install of Windows Server 2012 r2.  ArcGIS Server 10.3 and GeoEvent processor installed.  Any advice on where to look for the cause of this error would be greatly apprecitated!  Thanks!

  2015-05-21 13:15:17,043 | ERROR | Thread-45  | ConnectionHandler          | .lyra.internal.ConnectionHandler  125 | 243  - com.esri.ges.framework.messaging.jms-messaging - 10.3.0 | Failed to create connection cxn-160971 java.net.ConnectException: Connection refused: connect
    at java.net.DualStackPlainSocketImpl.connect0(Native Method)[:1.7.0_65]
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)[:1.7.0_65]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)[:1.7.0_65]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)[:1.7.0_65]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)[:1.7.0_65]
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)[:1.7.0_65]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)[:1.7.0_65]
    at java.net.Socket.connect(Socket.java:579)[:1.7.0_65]
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)[:1.7.0_65]
    at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:32)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:588)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:233)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:227)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at net.jodah.lyra.internal.RetryableResource.callWithRetries(RetryableResource.java:50)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:227)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:115)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at net.jodah.lyra.Connections.create(Connections.java:68)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at com.esri.ges.messaging.jms.AmqpMessagingFactory$ConnectionEstablisher.run(AmqpMessagingFactory.java:669)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]

0 Kudos
12 Replies
RJSunderman
Esri Regular Contributor

Sharon -

The log messages you provide above look to me like the Rabbit MQ message broker service (provided by ArcGIS Server) has failed or is not running.

  • Are you able to launch the GeoEvent Manager web application?
  • Can you create a simple TCP/Text input -> TCP/Text output GeoEvent Service?
  • If you send event data to the TCP input socket using GeoEvent Simulator, does the input's event count increment, but the 'In'/'Out' event count for the GeoEvent Service and event count on the output remain at zero (do not increment)?

There are some trouble shooting steps I can share with you that are found in the product tutorial for clustering. Technically even if you are configuring a Server site with only a single server machine, you are still running a cluster - just with only one machine.

Before we go into that, however, I'd like to get your feedback on the questions above.

Thanks -

RJ

0 Kudos
SharonMeier
New Contributor III

Hi, and thanks for the reply.  Yes, I can launch GeoEvent manager web application.   I have created input connector that is receiving data and incrementing but the service in/out counts are remaining at zero.  That's another issue I've been trying to track down, but based on your responses it sounds like it is related to this error message.   I will look at the information in your other log entry and try those steps. 

0 Kudos
RJSunderman
Esri Regular Contributor

If you suspect that ArcGIS Server has failed to correctly configure the RabbitMQ platform service:

  • In a Windows file explorer, browse to the ArcGIS Server configuration store.
    • The default location of the AGS config store is: C:\arcgisserver\config-store
    • The path to your configuration store is reported by Server Manager. Navigate to Site > Configuration Store to observe the path reported by Server Manager.

  • In your Windows file explorer, browse to the …\config-store\locks folder.
    • The only file you should find here should be a file named site.rlock

If you discover any files in …\config-store\locks whose names match the pattern machine-<MACHINE_NAME>.wlock or machine-<MACHINE_NAME>.wlock.info:

  • Stop the GeoEvent Windows Service
  • Stop the ArcGIS Server Windows Service

  • Examine the running tasks on the GIS Server
    • Force stop any processes still running that were launched from the ArcGIS Server Installation directory.
    • Click the illustration below to see a larger representation of the illustration.

Capture1.png

The illustration above only shows a couple of the processes frequently found to still be running when the RabbitMQ platform service failed to initialize correctly and the ArcGIS Server Windows Service has been shut down.

With AGS stopped, there should be no processes running whose command line path indicates they were launched from either:

-  C:\Program Files\ArcGIS\Server

-  C:\PROGRA~1\ArcGIS\Server

  • Back in your Windows file explorer, in the …\config-store\locks folder
    • Delete any *.wlock or *.wlock.info files you find.
    • Again, site.rlock is  the only lock file you should need to keep.

  • Restart ArcGIS Server Windows Service on your GIS Server.

Once you are able to launch and log-in to the ArcGIS Server Manager web application, go ahead and restart the GeoEvent Windows Service.

SharonMeier
New Contributor III

I've deleted the .wlock and .wlock.info files that were in my \config-store\locks folder.   Restarted ArcGIS Server service and GeoEvent service.  I'm still getting the error in my log file.   How can I confirm that RabbitMQ is running?

0 Kudos
RJSunderman
Esri Regular Contributor

Sharon -

I've created a branch for this discussion to place information on a diagnostic utility you can use to check the status of the RabbitMQ platform service in a separate thread.

Please check-out my post 'Diagnostic utility for checking the status of the RabbitMQ platform service'.

- RJ

0 Kudos
JavierDelgadillo
Esri Contributor

Sharon,

In order for this trick to work, you must stop the Windows services for GeoEvent and ArcGIS Server first.  Once they're both stopped, then you go and check the config folder for the *.wlock or *.wlock.info files and kill any processes that may still be running that show as being launch from C:\Program Files\ArcGIS\Server or  C:\PROGRA~1\ArcGIS\Server.  Once you've done both of those with the services not running, then restart ArcGIS Server followed by GeoEvent. 

This sequence usually rescues RabbitMQ for me, but the order is important.

Can you please confirm you tried doing the steps in the order mentioned above?

-Javier

SharonMeier
New Contributor III

Javier,

Yes, I tried the steps in the order listed.  Just tried them a second time, again in the order listed, and am getting the same result.  It appears that RabbitMQ is not running.

0 Kudos
SharonMeier
New Contributor III

Javier,  tried the steps again yesterday.  The third time it worked.

0 Kudos
JavierDelgadillo
Esri Contributor

Good to hear.  Sorry it took so many tries.

-Javier

0 Kudos