Feature Count limit in a for loop?

ShaunLangley — Wed, 08 Nov 2023 20:15:03 GMT

Update: I got an error return while exploring an unrelated issue. It appears there is a recordCount parameter being passed that I did not set.

The default on the feature service is 10k records.

Any ideas how to override this in my expression?

Original Post

I'm running into some strange behavior with my data expression. I'm trying to preprocess some data before it goes to my dashboard to reduce the data load. However, I'm only now realizing that the for loop in my code (shown below) is terminating after 1000 records when there are more than 166k records in the input dataset. I thought it might be a limit on the service, but it appears that the service is set to 10k. Certainly l could foresee an issue hitting that limit, but in this case, the loop is terminating even before that point. Does anyone see an issue in my code or has experienced this in their own work?

Context: I'm reading data from an ArcGIS Monitor feature service. There is too much data for Dashboard to have to parse through so I'm trying to reduce the volume. Note: I've masked the portal url and id due to an NDA with my client. It shouldn't affect readability.

// Reference layer using the FeatureSetByPortalItem() function. var portal = Portal('https://---.com/arcgis') var collectionId = '--' var fs = FeatureSetByPortalItem( portal, collectionId, 5, ['component_id', 'name', 'value', 'observed_at'] ); // return only records within the last day, the max time period supported var time_delta = DateAdd(Now(), -24, 'hours'); var filtered = OrderBy( Filter(fs, "observed_at > _delta and name = 'Requests Received'"), "component_id asc" ) // get services table to pull names for the component ids var svs = FeatureSetByPortalItem( portal, collectionId, 4, ['*'], false ); // build the output schema to write to var output = { fields: [ { name: "component_id", type: "esriFieldTypeInteger" }, { name: "component", type: "esriFieldTypeString" }, { name: "metric", type: "esriFieldTypeString" }, { name: "observed_at", type: "esriFieldTypeDate" }, { name: "total", type: "esriFieldTypeInteger" } ], 'geometryType': '', 'features': [] }; // loop over each record in the filtered dataset and write the name and metrics to the output var feat; var featureCount for (var a in filtered) { //var s = First(Filter(svs, 'id = ' + a.component_id)) feat = { attributes: { component_id: a.component_id, component: '', metric: a.name, observed_at: Number(a.observed_at), total: a.value } } Push(output.features, feat) } var fs = FeatureSet(Text(output)) var grouped = GroupBy( fs, ["component_id", "component", "observed_at"], { name: "Total", expression: "total", statistic: "SUM" } ) Console("input count: " + Count(filtered)) Console("processed count: " + featureCount) Console("output count: " + Count(fs)) Console("services count: " +Count(svs)) Console("grouped count: " + Count(grouped)) return grouped //return filtered

Edit: here is the output from the console showing the counts

input count: 166894
processed count: 1000
output count: 1000
services count: 94
grouped count: 1000

topic Feature Count limit in a for loop? in Developers Questions

Feature Count limit in a for loop?