ArcGIS Geo Event Server Email Output Stopped Working

[JamesMadden1]

We have a GeoEvent Service that sends an email based on a series of geofences.  The service gets its input from polling a feature service and sends emails when a point enters a geofence.  The email output worked for about a month and then stopped sending emails the other day.  I went into the logs and didn't see any relevant errors.  The service's input and output counts increase but the email's output count isn't increasing.  Wondering if anybody has encountered similar issues or knows of potential remedies?  We are sending a fair amount of emails, so I wouldn't be surprised if the output is getting flagged as spam.  The issue does resolve itself if I create a new email output and replace the existing one but I don't really want to keep regenerating output connectors every few weeks.

[Anonymous User]

That's great James Madden‌

You might be able to get reason from event log from the server, why it is restarted as well.

[EricIronside]

Hey James Madden‌,

It is not recommended to automatically reboot GeoEvent on a periodic basis. If you can avoid that, it would probably be a good thing. If you do have to reboot, as a safety precaution, you should turn off all of your GeoEvent inputs, services, and outputs prior to stopping the GeoEvent Windows service and rebooting. Once the machine returns to life, restart your GeoEvent outputs, services, and inputs (in that order). Doing this ensures that things start up in an orderly fashion and all dependencies are accounted for.

If you are in a situation where you are forced to automatically reboot the machines on a periodic basis. I would implement a python script to run on shutdown and startup that performs the stop/start operations for you.  You can find instructions for how to do that in Andy Ommen's article Scripting Tasks Using the GeoEvent Admin API.

Best regards,

Eric

[JamesMadden1]

Our servers had to get rebooted over the weekend and I noticed GeoEvent started to log a number of errors once the reboot happened.  The reboot only affected our dev environment which doesn't have an email service.  The error logs were mostly pointing at the message bus service.  I was able to resolve the issue by restarting the message bus a few times.  I have heard others describe similar issues with 10.6.1 and it seems like the common resolution is to restart services or GeoEvent itself.  I have read some threads that recommend starting geoevent and the gateway with a delayed state (delayed start).  It seems like the long term resolution might be to ensure our environment conforms to ESRI's recommendations and standards.  I don't believe we can prevent the servers from getting rebooted since they are subject to monthly service updates but we should be able run a Python script that restarts services on reboot.

[JamesMadden1]

Wanted to chime in on this thread as we've found a suitable resolution.  We have a number of outputs that stop working after rebooting our server.  We limit our server maintenance windows but we still have about one reboot per month.  The reboot causes certain processes to stop working once GeoEvent Server is back up.  We've been able to keep our outputs working by running two scripts when a reboot is initialized.  The first script basically turns off a service's outputs, inputs and self in that order.  That script is run prior to restarting the server.  The second script turns everything back on once the reboot has finished.   We haven't identified all the outputs that break when a reboot is initialized but these steps do keep the email output working.  The scripts themselves are created in Python and run from Task Scheduler (start services) and the Group Policy Manager (shutdown, stop services).  The shutdown script runs from a .BAT file.

[JamesMadden1]

So our services eventually broke with the steps outlined in my previous comment.  Everything was working but our email service stopped working after our seventh reboot.  I did some additional digging and noticed I hadn't installed the ArcGIS Server Security 2019 Update 2 Patch.  I went ahead and installed the patch and reverted all of our startup routines to their default settings.  The email services continued to work at this point but our GeoEvent logs were blowing-up with rabbitmq errors.  I made yet another update to our startup routine so that ArcGIS Server waits until 2 minutes after startup before beginning its process.  The gateway and geoevent then wait an additional two minutes each before beginning their respective processes.  This results in about a six minute startup routine and everything appears to be working.  Important thing to point out here is that we are no longer stopping services at shutdown and restarting at reboot.