topic Re: Error log entry: in ArcGIS GeoEvent Server Questions

Error log entry:

SharonMeier — Thu, 21 May 2015 20:11:08 GMT

The following message is appearing in my log file. Not sure where to begin troubleshooting. This is a new server, clean install of Windows Server 2012 r2. ArcGIS Server 10.3 and GeoEvent processor installed. Any advice on where to look for the cause of this error would be greatly apprecitated! Thanks!

2015-05-21 13:15:17,043 | ERROR | Thread-45 | ConnectionHandler | .lyra.internal.ConnectionHandler 125 | 243 - com.esri.ges.framework.messaging.jms-messaging - 10.3.0 | Failed to create connection cxn-160971 java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method)[:1.7.0_65] at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)[:1.7.0_65] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)[:1.7.0_65] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)[:1.7.0_65] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)[:1.7.0_65] at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)[:1.7.0_65] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)[:1.7.0_65] at java.net.Socket.connect(Socket.java:579)[:1.7.0_65] at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)[:1.7.0_65] at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:32)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:588)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:233)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:227)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at net.jodah.lyra.internal.RetryableResource.callWithRetries(RetryableResource.java:50)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:227)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:115)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at net.jodah.lyra.Connections.create(Connections.java:68)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at com.esri.ges.messaging.jms.AmqpMessagingFactory$ConnectionEstablisher.run(AmqpMessagingFactory.java:669)[243:com.esri.ges.framework.messaging.jms-messaging:10.3.0] at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]

Re: Error log entry:

RJSunderman — Thu, 21 May 2015 22:25:33 GMT

Sharon -

The log messages you provide above look to me like the Rabbit MQ message broker service (provided by ArcGIS Server) has failed or is not running.

Are you able to launch the GeoEvent Manager web application?
Can you create a simple TCP/Text input -> TCP/Text output GeoEvent Service?
If you send event data to the TCP input socket using GeoEvent Simulator, does the input's event count increment, but the 'In'/'Out' event count for the GeoEvent Service and event count on the output remain at zero (do not increment)?

There are some trouble shooting steps I can share with you that are found in the product tutorial for clustering. Technically even if you are configuring a Server site with only a single server machine, you are still running a cluster - just with only one machine.

Before we go into that, however, I'd like to get your feedback on the questions above.

Thanks -

Re: Error log entry:

RJSunderman — Thu, 21 May 2015 23:52:19 GMT

If you suspect that ArcGIS Server has failed to correctly configure the RabbitMQ platform service:

In a Windows file explorer, browse to the ArcGIS Server configuration store.
- The default location of the AGS config store is: C:\arcgisserver\config-store
- The path to your configuration store is reported by Server Manager. Navigate to Site > Configuration Store to observe the path reported by Server Manager.

In your Windows file explorer, browse to the …\config-store\locks folder.
- The only file you should find here should be a file named site.rlock

If you discover any files in …\config-store\locks whose names match the pattern machine-<MACHINE_NAME>.wlock or machine-<MACHINE_NAME>.wlock.info:

Stop the GeoEvent Windows Service
Stop the ArcGIS Server Windows Service

Examine the running tasks on the GIS Server
- Force stop any processes still running that were launched from the ArcGIS Server Installation directory.
- Click the illustration below to see a larger representation of the illustration.

The illustration above only shows a couple of the processes frequently found to still be running when the RabbitMQ platform service failed to initialize correctly and the ArcGIS Server Windows Service has been shut down.

With AGS stopped, there should be no processes running whose command line path indicates they were launched from either:

- C:\Program Files\ArcGIS\Server

- C:\PROGRA~1\ArcGIS\Server

Back in your Windows file explorer, in the …\config-store\locks folder
- Delete any *.wlock or *.wlock.info files you find.
- Again, site.rlock is the only lock file you should need to keep.

Restart ArcGIS Server Windows Service on your GIS Server.

Once you are able to launch and log-in to the ArcGIS Server Manager web application, go ahead and restart the GeoEvent Windows Service.

Re: Error log entry:

SharonMeier — Fri, 22 May 2015 13:11:38 GMT

Hi, and thanks for the reply. Yes, I can launch GeoEvent manager web application. I have created input connector that is receiving data and incrementing but the service in/out counts are remaining at zero. That's another issue I've been trying to track down, but based on your responses it sounds like it is related to this error message. I will look at the information in your other log entry and try those steps.

Re: Error log entry:

SharonMeier — Fri, 22 May 2015 15:44:09 GMT

I've deleted the .wlock and .wlock.info files that were in my \config-store\locks folder. Restarted ArcGIS Server service and GeoEvent service. I'm still getting the error in my log file. How can I confirm that RabbitMQ is running?

Re: Error log entry:

RJSunderman — Fri, 22 May 2015 23:53:02 GMT

Sharon -

I've created a branch for this discussion to place information on a diagnostic utility you can use to check the status of the RabbitMQ platform service in a separate thread.

Please check-out my post 'Diagnostic utility for checking the status of the RabbitMQ platform service'.

- RJ

Re: Error log entry:

JavierDelgadillo — Sat, 23 May 2015 00:35:14 GMT

Sharon,

In order for this trick to work, you must stop the Windows services for GeoEvent and ArcGIS Server first. Once they're both stopped, then you go and check the config folder for the *.wlock or *.wlock.info files and kill any processes that may still be running that show as being launch from C:\Program Files\ArcGIS\Server or C:\PROGRA~1\ArcGIS\Server. Once you've done both of those with the services not running, then restart ArcGIS Server followed by GeoEvent.

This sequence usually rescues RabbitMQ for me, but the order is important.

Can you please confirm you tried doing the steps in the order mentioned above?

-Javier

Re: Error log entry:

SharonMeier — Tue, 26 May 2015 13:57:40 GMT

Javier,

Yes, I tried the steps in the order listed. Just tried them a second time, again in the order listed, and am getting the same result. It appears that RabbitMQ is not running.

Re: Error log entry:

SharonMeier — Wed, 27 May 2015 12:27:25 GMT

Javier, tried the steps again yesterday. The third time it worked.

Re: Error log entry:

JavierDelgadillo — Wed, 27 May 2015 21:16:33 GMT

Good to hear. Sorry it took so many tries.

-Javier

Re: Error log entry:

AdamDunlap — Wed, 03 Jun 2015 21:29:14 GMT

RJ -

Thanks for the post. I have 10.3 GeoEvent Service and it seems to have the 3 extra files...but they go away after a period of time...only to re-emerge in ~15 seconds. I can see input data from a server in Birmingham, but I can't get the output or GeoEvent Service to create any "Out".

I have a net.jodah.lyra.internal.ConnectionHandler error in my Logs file and when I ran your diagnostic tool, I get the following about 'bad clustering'. Any thoughts?

Re: Error log entry:

RJSunderman — Thu, 04 Jun 2015 23:34:42 GMT

Hello Adam -

My understanding is that finding *.wlock or *.wlock.info files in your ArcGIS Server configuration store is not, in itself, a problem. Likewise finding the erl.exe and epmd.exe processes associated with RabbitMQ running is not a problem.

The problem is that the lock files and processes persist after the Windows Service used to run your ArcGIS Server instance has been stopped. Server is using the lock files to indicate to different processes that it is writing to some portion of its configuration ... and it should't be if its Windows Service is not running.

I have not observed the lock files periodically go away and come back - that might indicate that Server is trying (and failing) to configure the platform service. Did you try the remedy outlined by Javier above? (My apologies if you did, it wasn't clear to me from your post.)

The output you've illustrated from the diagnostic tool suggests to me that if you were to try to status the platform service via its administrative endpoint (refer to notes below my post with the diagnostic tool) you would discover that the platform service either was not configured or could not be started.

Sorry, but I think you are going to have to contact Esri Technical Support. The RabbitMQ platform service is configured and instantiated when ArcGIS Server is installed. If the configuration or launch of the platform service fails - GeoEvent will simply not work.

- RJ

Re: Error log entry:

YousefQuran — Wed, 16 Dec 2015 09:45:02 GMT

Javier,

Many thanks. works fine with us from the first try.

Regards,

Yusuf Ali