Diagnostic utility for checking the status of the RabbitMQ platform service

[RJSunderman]

There is a very basic utility bundled with the Clustering tutorial which will request a status from the RabbitMQ platform service, attempt to connect, and send a message to see if the specified host is able to receive messages.

Please download and unzip the attached CheckRabbitMQ.zip to a local folder on the server on which you have installed ArcGIS Server and the GeoEvent Extension. Open a Windows command shell and change directory to the folder containing the files you extracted from the ZIP archive.

In your Windows command shell, execute the commands illustrated below:

You should see the following usage instructions:

• Enter the Machine name, Port, Admin user and admin password, separated by a space.
• (For example: “machine 6080 admin admin_pass):

I tend to specify a fully qualified server name and domain (e.g. machine_name.esri.com) - and because my ArcGIS Server is configured to allow HTTPS connections - I specify port 6443. You could also specify localhost and port 6080 if your ArcGIS Server is only configured for HTTP connections.

If the utility is able to retrieve a status from the RabbitMQ platform service, connect, and receive a test message, you should see messages like the ones illustrated below:

Update March 2017:

The diagnostic utility rabbitmqdiag-1.0.jar is throwing a com.esri.arcgis.discovery exception when run against the 10.5 release of ArcGIS Server. Users have reported the same exception being thrown when the utility is run against the 10.4.1 release. The GeoEvent product team does not have the source code for this utility. It was provided as an elementary diagnostic to the GeoEvent team when RabbitMQ was first being integrated. It is not actually part of either the ArcGIS Server product or the GeoEvent Server product and is not being supported. If a new utility can be developed for the 10.5 product release comments will be added to this thread; otherwise please assume that the utility will only work for a 10.3.x product deployment.

[RJSunderman]

There is an undocumented administration REST endpoint you can reach to check the status of the RabbitMQ platform service. I should emphasize that we did not document this endpoint because its accessibility and interface is subject to change. The RabbitMQ platform service was introduced at 10.3 and GeoEvent is currently the only component using the platform service. This could all change with future releases of ArcGIS for Server - but the information below should be valid for the 10.3 and 10.3.1  product releases.

In a browser window, navigate to the ArcGIS Server Administrator Directory and log-in as the primary site administrator.

Once you have logged-in, enter the following URL into your browser to navigate to the platform services administrative endpoint:

• https://my-host:6443/arcgis/admin/system/platformservices/?f=pjson

A browser plug-in will make the JSON presented at this endpoint easier to read, but is not required. The JSON you see should contain the fields and values in the following illustration. (For security considerations, the actual id, user, and password files have been obliterated in the illustration.)

You should be able to copy/paste the “id” from the platform services endpoint to build a new URL which will take you to the actual platform service:

• https://my-host:6443/arcgis/admin/system/platformservices/ <id> /status?f=pjson

The JSON presented to you from this status query should identify whether or not the platform service is currently configured and on which servers the platform service is running:

You can start, stop, and query status using the administrative endpoint above:

• https://my-host:6443/arcgis/admin/system/platformservices/ <id> /start?f=pjson
• https://my-host:6443/arcgis/admin/system/platformservices/ <id> /stop?f=pjson
• https://my-host:6443/arcgis/admin/system/platformservices/ <id> /status?f=pjson

[SharonMeier]

Good morning,

I used the the REST endpoint and got the following:

{

  "Configuredstate":  "STARTED",

  "details":  [{

      "machine" :   machine name,

      "realtimeState":  "STOPPED"

  }],

"Status": "success"

}

The CheckRabbitMQ utility shows a running status of "STOPPED" and  Connection failed/refused.

[SharonMeier]

I went through the process of stopping and restarting the services and deleting the wlock files again.  Third time seemed to do the trick. 

[JerryStafurik]

Hi RJ,

Is there a way to check the RabbitMQ queues -- something similar to the backlog monitor in 10.2?  Also, is there a way to access the RabbitMQ management console?

Thanks,

  Jerry

[RJSunderman]

Hello Jerry -

Yes, there is a RabbitMQ management console you can access to check the internal queues, but we normally don't advertise how. The administrative endpoints are undocumented because our dependency on the RabbitMQ platform service is considered an implementation detail. The GeoEvent product team could change this at any release, although it did not change for 10.4 and is looking (for now) like it won't be changing for the 10.4.1 release.

I've been told that enabling the plug-in increases the overall memory footprint of the product, so you should probably only leave it enabled when you plan on actually using it, disabling it on a production system.

Finally, you can really muck things up from within the management console, so if you do enable it, be careful not to make any changes to the exchanges or message queue bindings.

Here's how to enable and access the management console:

• If RabbitMQ is running, you must first stop it
  • Log-out of GeoEvent Manager and stop the GeoEvent Windows Service
  • Log-in to the ArcGIS Server Administrator Directory:  https://<server>:6443/arcgis/admin/login
    • Navigate to the platform services endpoint:  http://<server>:6080/arcgis/admin/system/platformservices?f=json
    • This is an undocumented REST endpoint within the Server Admin API
  • Note the <id>, <user> and <password> for the RabbitMQ Message Bus displayed on the platform services page
  • Navigate to the following endpoint to stop the RabbitMQ platform service
    • http://<server>:6080/arcgis/admin/system/platformservices/<id>/stop?f=json
    • You can replace ‘stop’ in the URL above with either ‘status’ or ‘start’ to start, stop, and status the service
      

• In a Windows File browser, open the folder C:\Users\…\AppData\Roaming\RabbitMQ on the server machine
  for the user account used to run the ArcGIS Server and GeoEvent Windows Services.
  • Create a file named:  enabled_plugins
    • The filename has no extensions (make sure Windows isn’t hiding a .txt extension from you)
  • Edit the file to contain a single line of text:  [rabbitmq_management].
    • Note that the ‘.’ Is part of the line of text you enter into the file

• Use the ArcGIS Server administrative endpoint to start the RabbitMQ platform service
  • http://<server>:6080/arcgis/admin/system/platformservices/<id>/start?f=json
  • Refer to:  http://mediawiki/index.php/RabbitMQ_Integration#Enable_RabbitMQ_Management_plugin

• Restart your GeoEvent Windows Service
  • You do not need to be logged-in to GeoEvent Manager, but GeoEvent needs to be running

• Browse to the following URL to log-in to the RabbitMQ management console (using <user> and <password> noted above)
  • https://<server>:27274

From here you should be able to click on ‘Exchanges’ to review the exchanges managing communication for the GeoEvent ‘default’ cluster. You should see several exchanges which identify GeoEvent  (e.g.  geoevent.<cluster>.<type>:<guid> is how they appear in the 10.4 release).

If you log-in to the GeoEvent administrative API (https://<server>:6143/geoevent/admin) you can click to list the inputs you have running, for example, and look at the GUID assigned as the Name for an input. Back in the RabbitMQ management console you can click on ‘Queues’ and identify the queue associated with a particular GeoEvent input.

Good luck, and hope this information is helpful -

RJ

[GREGREECE]

rsunderman-esristaff‌

I upgraded both server and geoevent ext to 10.4.1 and now I cant get my services to edit the feature service. the data is coming in but thats it. it doesnt even show it getting to the 'geoevent service' part.

I followed all the steps here:Error log entry: 

i installed a fresh geoevent, set up everything all over again. deleted the data folders you mention, same problem.

i also ran this tool and the tool will not even run. try admin and user. 

i tried to navigate to this: http://localhost:6080/arcgis/admin/system/platformservices?f=json 

It returns this: {"platformservices":[],"status":"success"}

I went here C:\Users\…\AppData\Roaming\RabbitMQ

and this folder not there. or anywhere from what i can tell.

I am curious that since i upgraded to 10.4.1 from 10.2.2, that maybe the RabbitMQ did not get installed?

thanks in advance!

[RJSunderman]

Hello Greg -

From what you've indicated, you have faithfully tried everything I know to try to investigate the ArcGIS Server platform service which provides GeoEvent its message broker (e.g. RabbitMQ).  Your indications suggest to me that the platform service did not get installed / upgraded ... you may have to uninstall GeoEvent, uninstall ArcGIS Server and attempt to re-install the product (versus attempting an in-place product upgrade).

I've contacted Esri Technical Support - someone should be reaching out to you soon.

Unfortunately the RabbitMQ platform service is somewhat of a black box from a GeoEvent perspective. It it's healthy and running, GeoEvent works - if not then GeoEvent won't work at all.  I've contacted Tech Support because this is an area of the ArcGIS Server platform that I cannot offer further advice on.

Good Luck -

RJ

[WillHughes2]

Any reason why the realtimeState status would be stopped and not able to start?

Apache Spark Platform Service. Used for GeoAnalytics Ext. 

Status {

    "configuredState": "STARTED",

    "details": [{

        "machine": "P1VZ-AA-WXAQT.GRAY.HRTD.PEM",

        "realtimeState": "STOPPED"

    }]

}