Weird Error August 13th 3pm * Uncaught Exception in KS!

[Teresa_Blader]

I have an API feed and a real-time analytic pushing data to a feature layer, and sending an email with a new incident alert. I was checking in on it today after a suspiciously low number of alert emails in the last 24 hours. There's usually 2 or 3 a day.

My real time analytic has been happily running so I thought, then again it seems it's behavior has completely changed as of August 13th at 3pm CST. Real-time analytic missed a new incident from 4am this morning, this means the API feed had the 4am incident but it wasn't pushed into the feature layer via the real-time analytic. The feed itself appear to be just fine.

Just wondering what this error might mean and why the real-time analytics logs look very different since. Is my real-time analytic ok?? It seems like it might not be.

* Uncaught Exception in KS! Thread: 9cf14bc1a39347ababd9d71164d23056-1723226566982-debd474a-6b55-4fe8-aa88-9740d7033605-StreamThread-1-40

Following this error, there's no longer the normal *Upserting count features message, until I stopped and restarted the analytic this afternoon. 

When I restarted this afternoon I keep getting this message I've never seen before:

* All output items are in the same data store as the cluster which is: None

[JeffSilberberg]

@ArmstKP 

   Not to my knowledge.  Have always been told to do a STOP / START manually. 

    But maybe you could use something like Auto-Hotkey to do it.  
     https://www.autohotkey.com/

[ArmstKP]

Yesterday, I received the Uncaught Exception in KS! Thread for another RT analytic and it stopped outputting data and I didn't notice it until now (one day of data lost).  To fix the issue - I stopped the analytic and started it.

[JeffSilberberg]

@armarmmer 

"(one day of data lost)"  As I think I said previously.  Inject a Transaction into the Feed with an Unquie EventType like "WatchDog" or something and monitor that function to be sure the Feed accepts the transaction.   Then towards the end of your Analytic filter off the Event and use an HTTP post node to send the transaction back to a monitoring point.  Set up alerts from the Monitoring point to tell you when there's an issue.  

Note: I give my WatchDogs the fixed value Lat & Lon of the customer HQ so that the transaction can flow through the Analytic. 

[ArmstKP]

Another update.  All three of our real time analytics received the "Uncaught Exception in KS! Thread" error at the same time yesterday (12:22pm).  All of them stopped outputting data.

[Teresa_Blader]

Just want to report that there was an error today (stop and restart brough RT analytic back online)

* Uncaught Exception in KS! Thread: e54da2ef4b7743cfbb30151e842c4552-1736186872657-47620428-2c2b-4a46-b725-f74b3da43b60-StreamThread-1-52

[KAMPRB]

@Teresa_Blader  I received another error just like this the other day.  I had to manually stop and restart to get the analytic to work again.  Lost one day of data, thus making our maps to our field screws incorrect for the next two weeks.  Another Kafka outage?

[Teresa_Blader]

In my case, after thorough Esri support investigation - there were some apparent oddities in my real-time analytic logs with regenerating tokens every 25 minutes every 2-10 minutes. And some issues with how Kafka stores interim information. All of which came down to some updates/future work arounds the production team is working on. Nothing intrinsic to my setup per se. 

that being said, it was recommended to me here to issue a support ticket to allow them to investigate. So I pass along that recommendation. 

there’s seems to be two situations that shut down the analytics - one is where all analytics have a shutdown error at the same time and some just impact certain analytics at certain times.

either way I have adopted a regular strategy of stopping and restarting feeds. I find I less errors and unexpected shutoffs happen. I time this with my every other week update of my power automate survey123 connection credentials refresh.

Hope that helps!

[JeffSilberberg]

Good afternoon, 

     I have not been watching the Velocity Logs closely, as it's a very time-consuming process to check them all.  But the Watchdog I have running alerted me on Monday, 4/14 at 13:16 EDT that transactions were not processing, and the issue cleared at 13:31. In that window, we had about 2K transactions queued, and all processed when Velocity came back up.