ArcGIS Geo Event Server Email Output Stopped Working

JamesMadden1 · ‎01-21-2020

We have a GeoEvent Service that sends an email based on a series of geofences. The service gets its input from polling a feature service and sends emails when a point enters a geofence. The email output worked for about a month and then stopped sending emails the other day. I went into the logs and didn't see any relevant errors. The service's input and output counts increase but the email's output count isn't increasing. Wondering if anybody has encountered similar issues or knows of potential remedies? We are sending a fair amount of emails, so I wouldn't be surprised if the output is getting flagged as spam. The issue does resolve itself if I create a new email output and replace the existing one but I don't really want to keep regenerating output connectors every few weeks.

Anonymous User · ‎01-21-2020

Hi James Madden‌,

I have similar issue with different output connector (write to STBDS) , (from services.msc)

After I restarted the geoevent service, it resumes working automatically, how about you try like this?

JamesMadden1 · ‎01-22-2020

Than, I appreciate the response. Are you aware of what the underlying issue was with the STBDS connector? I understand reconfiguring the service or restarting it might provide a temporary resolution but my organization will have to seriously consider our use cases for GeoEvent Server if these techniques are the approved way to go about fixing defects. Again, I appreciate you taking the time to respond.

Anonymous User · ‎01-22-2020

Hi James Madden‌,

For underlying issue with STBDS, I didn't manage to figure out at that time. Same as your situation that time, I am quite crazy playing with Geoevent. (Last year).

As far as I remembered, I found a lot of kafka error log like zookeeper connection failed or something. (you can get the log from geoevent installed folder, logs) I think.

Or

Turn on the log setting at manager => http..... :6143/geoevent/manager/log.html, click settings , log level to debug.

And then delete all logs to be sure all logs are clear,

Stop all other services. (Make sure you don't touch the services you got issue)

And wait until you see the log.

You might get some useful log that way.

Another way how I troubleshoot for that kind of situation is by adding the additional output connector

Like for your flow -> WIFI_GEOTAGGER to WIFI geofence email , (add additional output connector like json-output , connect to WIFI_GEOTAGGER) and log down the output of the process.

Sometime the output from the process is not irrelevant to process by the output connector (for example, null value)

JamesMadden1 · ‎01-23-2020

Than,

I added a new email output and configured it with the exact same properties as the original. I replaced the original one with the updated output and the service is now sending emails. I have placed the original email connector into a new service and it doesn't appear to be working. I'm going to continue trying to resolve the issue as we need a stable output that doesn't have to get reconfigured.

Do you have any suggestions based on the updated information I have provided? I think the issue might be related to the Kafka service but my overall knowledge of GeoEvent Server is somewhat limited at this time. I can tell you that the connector stopped sending emails on a Saturday and there shouldn't have been any changes made at that time.

Also, I tried checking the logs with the setting set to debug. There was an awful lot of data to sift through in the logs but I wasn't able to locate anything directly related to the email connector.

JamesMadden1 · ‎01-22-2020

I've tried restarting the service a couple times and that doesn't seem to work.

JamesMadden1 · ‎01-22-2020

I went into the Admin page for GeoEvent and I noticed my flow doesn't list the connectors in the order they are run. The flow goes from GEOFENCE_POLL_LAYER --> WIFI Geofence --> WIFI_GEOTAGGER --> WIFI Geofence Email. The flow that the service shows is provided below. Could this be causing an issue? Again, the email worked for about a month and then stopped abruptly a couple days ago.

Flow:

From: WIFI_GEOTAGGER
To:
- WIFI Geofence Email
From: GEOFENCE_POLL_LAYER
To:
- WIFI GEOFENCE
From: WIFI GEOFENCE
To:
- WIFI_GEOTAGGER

JamesMadden1 · ‎01-24-2020

So I did notice something in the logs that might provide a clue. I created of copy of the original email output and added that to my geofencing service. I then created a test service with the original email. The geofencing service is sending the emails out and the test service is still not working. The pic below shows what the logs return when I start all the inputs, outputs and services for the two services. Notice how there are two "SUCCESS" messages. Everything below each success message pertains to a specific service. The logs at the bottom (bellow the second SUCCESS message) all pertain to the geofencing service that is working correctly. The test service logs are all nested below the SUCCESS message at the top. Now notice how the service that is working shows four logs that state "Route..... started and consuming from direct://". The test service only shows two of those logs. These logs belong to the logger "org.apache.camel.blueprint.BlueprintCamelContext". Could the error be related to this discrepancy?

JamesMadden1 · ‎01-24-2020

Noticed another thing. Our actual box where GeoEvent is installed got restarted at 8:45AM on 1/18/2020. Our service sent its last email at 8:09AM on the same day.

JamesMadden1 · ‎01-24-2020

So it appears we have identified what is causing our email service to stop working. Our servers get rebooted from time to time for various maintenance and patching. The reboot causes certain processes in GeoEvent to stop working. We currently know how to resolve the issue but we don't know how to prevent it. We have a failover process in place though I believe that requires us to make manual updates to our enterprise config for various components of our infrastructure. We are working on setting up a truly HA environment and that will hopefully prevent this issue from surfacing in the future. That said, I would still appreciate any helpful responses from the community.