GeoEvent outputs not working

ChrisBeyett · ‎11-03-2016

I've started a new question on your behalf so we can get a fresh look at what is happening in your system. If need be, I can help you get a Support case created and work with you to diagnose any problems. For now, let's look at the behavior you are seeing.

The original post can be found here:

https://community.esri.com/thread/175125

You are running a 10.4.1 Full Stack (formally known as a Web-GIS) with GeoEvent. All of the output connectors display a 0 count for the received Geoevents, and have been for a long period of time. Normally this still indicates a RabbitMQ or larger issue with the transport system behind GeoEvent. What we can do is start at the basic level here and see if we can determine the cause.

If you're able to, please attach a zipped copy of the GeoEvent log files. These can be found at C:\Program Files\ArcGIS\Server\GeoEvent\data\log. They are quite verbose, and I'm hoping they can show us what is occurring.

Also, have you noticed this as a recent error, or has it been present since the initial installation?

-Chris

JamesMadden1 · ‎01-23-2020

I encountered a similar issue with the send email output. The service shows the output increment but the output connector count is not increasing. I reconfigured the email connector with a new output. The new email has the exact same properties as the original and the service is working again. I placed the original email connector into a new test flow and it still doesn't work. I suspect the issue might be due to one of the controlling processes getting restarted after the output connector was running. Restarting the output doesn't seem to resolve the issue.

We are running 10.61 with plans to migrate to 10.7 in coming months.

JamesMadden1 · ‎03-11-2020

Wanted to chime in on this thread as we encountered similar issues and figured out a suitable resolution. We have a number of outputs that stop working after rebooting our server. We limit our server maintenance windows but we still have about one reboot per month. The reboot causes certain processes to stop working once GeoEvent Server is back up. We've been able to keep our outputs working by running two scripts when a reboot is initialized. The first script basically turns off a service's outputs, inputs and self in that order. That script is run prior to restarting the server. The second script turns everything back on once the reboot has finished. We haven't identified all the outputs that break when a reboot is initialized but these steps do keep the email output working. The scripts themselves are created in Python and run from Task Scheduler (start services) and the Group Policy Manager (shutdown, stop services). The shutdown script runs from a .BAT file.

JamesMadden1 · ‎03-16-2020

So our services eventually broke with the steps outlined in my previous comment. Everything was working but our email service stopped working after our seventh reboot. I did some additional digging and noticed I hadn't installed the ArcGIS Server Security 2019 Update 2 Patch. I went ahead and installed the patch and reverted all of our startup routines to their default settings. The email services continued to work at this point but our GeoEvent logs were blowing-up with rabbitmq errors. I made yet another update to our startup routine so that ArcGIS Server waits until 2 minutes after startup before beginning its process. The gateway and geoevent then wait an additional two minutes each before beginning their respective processes. This results in about a six minute startup routine and everything appears to be working. Important thing to point out here is that we are no longer stopping services at shutdown and restarting at reboot.