Weird Error August 13th 3pm * Uncaught Exception in KS!

[Teresa_Blader]

I have an API feed and a real-time analytic pushing data to a feature layer, and sending an email with a new incident alert. I was checking in on it today after a suspiciously low number of alert emails in the last 24 hours. There's usually 2 or 3 a day.

My real time analytic has been happily running so I thought, then again it seems it's behavior has completely changed as of August 13th at 3pm CST. Real-time analytic missed a new incident from 4am this morning, this means the API feed had the 4am incident but it wasn't pushed into the feature layer via the real-time analytic. The feed itself appear to be just fine.

Just wondering what this error might mean and why the real-time analytics logs look very different since. Is my real-time analytic ok?? It seems like it might not be.

* Uncaught Exception in KS! Thread: 9cf14bc1a39347ababd9d71164d23056-1723226566982-debd474a-6b55-4fe8-aa88-9740d7033605-StreamThread-1-40

Following this error, there's no longer the normal *Upserting count features message, until I stopped and restarted the analytic this afternoon. 

When I restarted this afternoon I keep getting this message I've never seen before:

* All output items are in the same data store as the cluster which is: None

[ClayDonaldsonSWCA]

Potentially related to this current incident? We have been running into similar issues with inserting data.

INC-001547: Asynchronous operations are slow to complete on all service endpoints.

[Teresa_Blader]

I am unhappy to announce this error happened across all of my real-time analytics at 3:14pm CST yesterday.

[JeffSilberberg]

Hope you have opened a ticket with support.  

[Teresa_Blader]

Not yet. My last ticket related to errors on all my real-time feeds, which is probably still open, that we also messaged about Jeff, came to a screeching halt when I was asked by Esri support to install software and do a bunch of testing on my own I don’t actually have the skills for.

Theres not really any documentation about what errors in ArcGiS Velocity mean so was just trying to poll first if anyone had an information. 

[JeffSilberberg]

@PeterNasuti 

[JeffSilberberg]

@TeresaBlader  @PeterNasuti 

Just thought I would add that I saw a similar error on Saturday that took two Analytics offline for a significant period of time. They seem to have recovered on their own but I am not entirely sure as the Logs seem sparse after this entry -- 

  * Uncaught Exception in KS! Thread: d9caed905ea141b78931fcf493ba3a02-1723055480341-dab4f2f0-b320-4dcc-8dbb-5957c7356704-StreamThread-1-51

[ArmstKP]

@Teresa_Blader @JeffSilberberg @ClayDonaldsonSWCA  As of this morning, I am now having this same issue with only one of our RT analytics.  We have not received data today after the time this error occurred.  We also do not see one log before the error happened, which is quite strange:

It would be great to create some sort of notification that an analytic is no longer outputting data, but is still running and not stopped.  By having something like this, one would not have days go by without noticing that data was never being outputted, instead one could create a case immediately and try to restart the analytic to see if that fixes the issue.  Anyone have any thoughts on how to go about building a notification like this?

In our organization, this analytic and the status values it calculates is visualized as a 14 day cycle status map to our field crews.  If data is lost one day and the status values aren't up to date, our status maps will not be accurate for the next 14 days...  Basically one day of data loss equals 14 days of not having correct data to our field staff.