topic Re: Weird Error August 13th 3pm * Uncaught Exception in KS! in ArcGIS Velocity Questions

Weird Error August 13th 3pm * Uncaught Exception in KS!

Teresa_Blader — Wed, 14 Aug 2024 22:16:19 GMT

I have an API feed and a real-time analytic pushing data to a feature layer, and sending an email with a new incident alert. I was checking in on it today after a suspiciously low number of alert emails in the last 24 hours. There's usually 2 or 3 a day.

My real time analytic has been happily running so I thought, then again it seems it's behavior has completely changed as of August 13th at 3pm CST. Real-time analytic missed a new incident from 4am this morning, this means the API feed had the 4am incident but it wasn't pushed into the feature layer via the real-time analytic. The feed itself appear to be just fine.

Just wondering what this error might mean and why the real-time analytics logs look very different since. Is my real-time analytic ok?? It seems like it might not be.

* Uncaught Exception in KS! Thread: 9cf14bc1a39347ababd9d71164d23056-1723226566982-debd474a-6b55-4fe8-aa88-9740d7033605-StreamThread-1-40

Following this error, there's no longer the normal *Upserting count features message, until I stopped and restarted the analytic this afternoon.

When I restarted this afternoon I keep getting this message I've never seen before:

* All output items are in the same data store as the cluster which is: None

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

ClayDonaldsonSWCA — Wed, 14 Aug 2024 22:29:35 GMT

Potentially related to this current incident? We have been running into similar issues with inserting data.

INC-001547: Asynchronous operations are slow to complete on all service endpoints.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

Teresa_Blader — Wed, 14 Aug 2024 22:31:14 GMT

I am unhappy to announce this error happened across all of my real-time analytics at 3:14pm CST yesterday.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

JeffSilberberg — Thu, 15 Aug 2024 14:23:27 GMT

Hope you have opened a ticket with support.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

Teresa_Blader — Thu, 15 Aug 2024 14:41:11 GMT

Not yet. My last ticket related to errors on all my real-time feeds, which is probably still open, that we also messaged about Jeff, came to a screeching halt when I was asked by Esri support to install software and do a bunch of testing on my own I don’t actually have the skills for.

Theres not really any documentation about what errors in ArcGiS Velocity mean so was just trying to poll first if anyone had an information.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

JeffSilberberg — Thu, 15 Aug 2024 14:48:29 GMT

@PeterNasuti

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

JeffSilberberg — Sun, 18 Aug 2024 14:38:24 GMT

@TeresaBlader @PeterNasuti

Just thought I would add that I saw a similar error on Saturday that took two Analytics offline for a significant period of time. They seem to have recovered on their own but I am not entirely sure as the Logs seem sparse after this entry --

* Uncaught Exception in KS! Thread: d9caed905ea141b78931fcf493ba3a02-1723055480341-dab4f2f0-b320-4dcc-8dbb-5957c7356704-StreamThread-1-51

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

ArmstKP — Sun, 29 Sep 2024 20:37:52 GMT

@Teresa_Blader @JeffSilberberg @ClayDonaldsonSWCA As of this morning, I am now having this same issue with only one of our RT analytics. We have not received data today after the time this error occurred. We also do not see one log before the error happened, which is quite strange.

UPDATE: The simple solution to start receiving data again was to stop the RT analytic and restart it.

It would be great to create some sort of notification that an analytic is no longer outputting data, but is still running and not stopped. By having something like this, one would not have days go by without noticing that data was never being outputted, instead one could create a case immediately and try to restart the analytic to see if that fixes the issue. Anyone have any thoughts on how to go about building a notification like this? Maybe an ArcGIS Notebook that runs every 15 minutes and looks at the logs of the analytic and if it contains the text , "Uncaught Exception in KS! Thread", stop the analytic and restart it?

In our organization, this analytic and the status values it calculates is visualized as a 14 day cycle status map to our field crews. If data is lost one day and the status values aren't up to date, our status maps will not be accurate for the next 14 days... Basically one day of data loss equals 14 days of not having correct data to our field staff.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

JeffSilberberg — Sun, 29 Sep 2024 18:57:04 GMT

Well,

I was told previously that "I believe it is related to some Kafka outages", and some new monitoring tools/restart procedures were being implemented.

For the data we collect and feed into Esri we have a couple of tools in place that, at least so far, minimize any loss of data. First, we process everything through a Queue and have to have a positive Ack before the entry is removed from the Transmitt Queue. But we also have a moving 45 day archive of all traffic. If the Transmitt Queue gets backed up we immediately send SMS notifications to a group of both Users and Admins.

Additionally, we submit a "Watchdog Transaction" every n minutes and accept that transaction back via an Analytic HTTP Post Node to our API. Again, if we miss a Watchdog coming back we generate SMS notifications. So we are watching that the Feed is accepting Transactions and that the Analytic is processing them.

Hope this helps.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

ArmstKP — Sun, 29 Sep 2024 22:56:43 GMT

@JeffSilberberg Is it just possible to stop the analytic and restart it automatically if a log contains the string "Uncaught Exception in KS!".? It seems like 9 out of 10 times this is the simple solution...

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

JeffSilberberg — Mon, 30 Sep 2024 02:26:33 GMT

@ArmstKP

Not to my knowledge. Have always been told to do a STOP / START manually.

But maybe you could use something like Auto-Hotkey to do it.
https://www.autohotkey.com/

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

ArmstKP — Fri, 04 Oct 2024 13:52:07 GMT

Yesterday, I received the Uncaught Exception in KS! Thread for another RT analytic and it stopped outputting data and I didn't notice it until now (one day of data lost). To fix the issue - I stopped the analytic and started it.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

JeffSilberberg — Mon, 07 Oct 2024 19:52:50 GMT

@armarmmer

"(one day of data lost)" As I think I said previously. Inject a Transaction into the Feed with an Unquie EventType like "WatchDog" or something and monitor that function to be sure the Feed accepts the transaction. Then towards the end of your Analytic filter off the Event and use an HTTP post node to send the transaction back to a monitoring point. Set up alerts from the Monitoring point to tell you when there's an issue.

Note: I give my WatchDogs the fixed value Lat & Lon of the customer HQ so that the transaction can flow through the Analytic.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

ArmstKP — Wed, 09 Oct 2024 13:16:59 GMT

Another update. All three of our real time analytics received the "Uncaught Exception in KS! Thread" error at the same time yesterday (12:22pm). All of them stopped outputting data.

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

Teresa_Blader — Fri, 10 Jan 2025 17:23:19 GMT

Just want to report that there was an error today (stop and restart brough RT analytic back online)

* Uncaught Exception in KS! Thread: e54da2ef4b7743cfbb30151e842c4552-1736186872657-47620428-2c2b-4a46-b725-f74b3da43b60-StreamThread-1-52

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

KAMPRB — Wed, 16 Apr 2025 12:59:49 GMT

@Teresa_Blader I received another error just like this the other day. I had to manually stop and restart to get the analytic to work again. Lost one day of data, thus making our maps to our field screws incorrect for the next two weeks. Another Kafka outage?

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

Teresa_Blader — Wed, 16 Apr 2025 13:34:42 GMT

In my case, after thorough Esri support investigation - there were some apparent oddities in my real-time analytic logs with regenerating tokens every 25 minutes every 2-10 minutes. And some issues with how Kafka stores interim information. All of which came down to some updates/future work arounds the production team is working on. Nothing intrinsic to my setup per se.

that being said, it was recommended to me here to issue a support ticket to allow them to investigate. So I pass along that recommendation.

there’s seems to be two situations that shut down the analytics - one is where all analytics have a shutdown error at the same time and some just impact certain analytics at certain times.

either way I have adopted a regular strategy of stopping and restarting feeds. I find I less errors and unexpected shutoffs happen. I time this with my every other week update of my power automate survey123 connection credentials refresh.

Hope that helps!

Re: Weird Error August 13th 3pm * Uncaught Exception in KS!

JeffSilberberg — Wed, 16 Apr 2025 15:44:09 GMT

Good afternoon,

I have not been watching the Velocity Logs closely, as it's a very time-consuming process to check them all. But the Watchdog I have running alerted me on Monday, 4/14 at 13:16 EDT that transactions were not processing, and the issue cleared at 13:31. In that window, we had about 2K transactions queued, and all processed when Velocity came back up.