Restrict number of records used in Summarize Attributes

Question

I am looking for any insight into if its possible to restrict the number of records in a track that are used to calculate summary statistics.

I have a Big Data Analytic (BDA) that is querying a feature layer output from a Real Time Analytic (RTA). The feature layer query in the BDA is for the last 3 minutes of data. Since the feeds in the RTA are polled every 5 seconds, I am expecting the number of records for each track to be queried in the BDA's input feature layer to be 36 --> (180 / 5). But I am finding when I run the summarize attributes, the number of joined or participating records in the summary calculation to fluctuate anywhere between 2 and 75 (the average is about 36 though). Reviewing the full output from the BDA, it appears that of the approx. 200 records, 65 had greater than 36 records participate in the summary calculations.

I want to restrict the number of records for each track that will participate the summary calculations so that only a maximum of 36 records will be used to calculate the summary. Is it possible to use the "Filter by Expression" tool using an arcade expression?

JeffSilberberg · Answer

@KevinPiraino2

Kevin,  I would check your joins you maybe be getting more results than you suspect if the leys are not unique primary for the inner join.   You also might check your sensor/reporting device. I recently had a case where I expected one report every ten seconds from every device, but found that the timing only applied to the "periodic" reports, and Telemetry events triggered by the onboard accelerometer and gyroscope sensors were intermixed and could result in many more transactions than expected in a given time window.  My solution was to filter down to the "periodic" reports in my RTA.

KevinPiraino2 · Answer

Jeff, thanks for the suggestion on the Track functions.

Upon further investigation on my issue, I noticed that in instances where the summary stats tool was using more than the expected 36 records, I found that the input feature layer was querying the correct time window for features, but that the output from the RTA (that is the input feature layer in the BDA) was writing more than the expected number of records per minute at times (upwards of 36 records in 1 minute). My assumption was that the RTA could only write to the output as quickly as the slowest input feed was providing data (polling every 5 seconds --> 60 / 5 == 12 records). Its possible my assumption on this aspect of the RTA could be incorrect, but I haven't done any additional testing to prove this theory.

I also wonder if the issue in the RTA I noticed is due to the fact I must poll my AVL API multiple times in a single RTA and join them consecutively in order to acquire all of the necessary attributes needed for my output. I don't fully understand the stateful vs. stateless aspect of RTAs or the consequences of using tools that function in a stateful manner, but its also possible that this is what is causing my issue as well.

JeffSilberberg · Answer

@KevinPiraino2

Kevin, a number of the recent posts here are similar to your issue. The general consensus has been to use the Arcade Track functions as the route to solving the issue.  I am working through a similar issue where the number of transactions in a five-minute window can vary greatly depending on the report's reason.

----  https://developers.arcgis.com/arcade/function-reference/track_functions/