topic Re: Errors in Velocity workflow: Feed, Realtime Analytic and output feature service layers NOT WORKING anymore in ArcGIS Velocity Questions

Errors in Velocity workflow: Feed, Realtime Analytic and output feature service layers NOT WORKING anymore

MattGeorge1 — Tue, 15 Aug 2023 19:05:36 GMT

Hello, I was hoping someone could spot something going on with my Velocity workflow, or perhaps something has changed since the end of July 2023 update. I’ve had no problems until recently with my workflows. I haven’t changed how things are setup either. I seem to be having problems along every step with the feed, Realtime analytic and feature service outputs. I’m not able to get anything to work.

My workflow is as follows.

I have a feed that runs in the Realtime analytic with two feature service outputs. Both outputs are Non spatiotemporal because they are used in public facing web maps. Never had a problem with this setup before but now I can’t get anything to work.

I have errors in the feature service data tables, sometimes I have no records, but sometimes if I refresh the feature service records will come back temporarily. Also, the layers would intermittently display the assets on the map sometimes for a few minutes and now they no longer display at all. The output feature services have always ran great the way they are setup I didn’t change anything. This started happening after the last update, but I don’t want to rule out that something else has gone sideways because there are a lot of pieces that go into the workflow.

Errors I’m seeing: Feed and Realtime analytic.

Feed: The feed is running at 0.4f/s with 1.75 on avg attributes/second. The feed ingests 54 total public works fleet trucks. I understand Velocity recommends no more than 2 events per second when running non spatiotemporal workflows. The recommendation or limitation is governed by our bronze level Velocity subscriptions at the 20K per year price point. At first, I thought maybe I’m ingesting too many attributes/features but I’m below the threshold, so I think that’s ruled out.

FEED SCREENSHOTS:

REALTIME ANALTYIC SCREENSHOTS BELOW:

FEATURE SERVICE OUTPUTS

The top FS output SnowFleet_outputComp is setup for Keep existing features and schema with add all new features, False for spatiotemporal feature layer.

The bottom FS output SnowFleet_outputCurrent is setup for Replace existing features and schema, keep only the latest feature for each Track ID value and again false for spatiotemporal.

The feature services are added to a webmap, they are styled using filters off Reportime to display current truck locations defined by the filters of different hours of time to display breadcrumbs. This setup has served us well.

I’m seeing errors everywhere. Also, in the webmap I’m using the Map Viewer, Not classic. And the feature service output layers are constantly updating but not showing anything on the map. The blue dot in the webmap layer flashes nonstop like a pinball machine gone haywire. Sometimes data shows up in the table other times not.

Is anyone else having a similar experience where an otherwise normal functioning workflow has stopped working how it should?

This all started at the end of July. I was able to get it to work on and off the last week or so, but now I can’t get anything to work.

Sometimes if I stop the feed and analytic and then start them back up, and I’ll see data show up on the map, but then it goes sideways shortly after.

Thanks for taking the time to try and make sense of this. I could really use some help to figure out how to resolve this. Or is there something going on out of my control that we are not aware of.

Matt

Re: Errors in Velocity workflow: Feed, Realtime Analytic and output feature service layers NOT WORKING anymore

JeffSilberberg — Tue, 15 Aug 2023 19:13:51 GMT

Matt,

Your overall description sounds very familiar to issues I have seen as well. Unfortunately, I can't offer much other than to suggest you open a case if you haven't already --

Re: Errors in Velocity workflow: Feed, Realtime Analytic and output feature service layers NOT WORKING anymore

JeffEismanGIS — Tue, 15 Aug 2023 19:17:47 GMT

@MattGeorge1 It looks like you have a lot going on, can you message me directly to set up a time for a screen share. I would like to look at your workflow and see if we can figure out what is happening.

Re: Errors in Velocity workflow: Feed, Realtime Analytic and output feature service layers NOT WORKING anymore

ArmstKP — Wed, 16 Aug 2023 20:35:29 GMT

@MattGeorge1 Matt, we are having very similar issues with ours. In speaking with tech support, they are investigating issues with clusters used by Velocity and/or event rates. Not much we can do either until there is a fix found. We have been basically without reliable data analytics for over a month. Very spendy 44k a year to have sidelined. Hang in there, although it is very difficult.

Advanced error logs:

Standard error logs:

Re: Errors in Velocity workflow: Feed, Realtime Analytic and output feature service layers NOT WORKING anymore

ArmstKP — Thu, 17 Aug 2023 18:39:18 GMT

@MattGeorge1 Today, as an example, we noticed that our data had gaps. Velocity was gathering information/vehicle tracks normally in the Realtime analytic. When I looked at the map and the output from the associated BDA, there were data gaps during the day. The BDA was working fine and updating statuses on the map when criteria was met, then there was a two hour stretch when it wasn't done. There were no errors in logs during this time and the BDA continued to run. After this stretch, there was an hour stretch where it began working again. Finally, there was again an hour stretch where it didn't process/change the status field...

What I am guessing the issue could be for us, is that we have had errors that our BDA can't locate our vehicle tracks hosted feature layer that was created as an output in the Realtime Analytic. So, therefore; when it can't find the vehicle tracks, it can't run our BDA correctly during those time.

Just wanted to let you know what I have seen today, to see if it mirrors what you see sometimes....

Re: Errors in Velocity workflow: Feed, Realtime Analytic and output feature service layers NOT WORKING anymore

PeterNasuti — Fri, 18 Aug 2023 13:04:30 GMT

Thanks @MattGeorge1 & @ArmstKP for your time today on separate calls.

For the case reported by @MattGeorge1 this was identified as being caused by Editor tracking being disabled on your output online hosted feature layers around the time of the last ArcGIS Online update. Once this was re-enabled by you, the analytic resumed successful processing.

For the case reported by @ArmstKP we are continuing to work together to isolate the cause of intermittent issues when Velocity is trying to write to the ArcGIS Online hosted feature layer REST API and receiving errors from ArcGIS Online. Investigation is ongoing and we will continue working directly to find a resolution.

For our Velocity users, please continue to log a support case if any issues are encountered with the product. Additionally, from the real-time product team we encourage posts to ask questions or present issues to engage with the real-time product team here on the Esri Community space so that we can ensure your success. Please also feel welcome to direct message on Esri Community myself, @JeffEismanGIS @Yini_Song or other Esri representatives that you see active in the Velocity and GeoEvent space. Thanks!