topic Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit? in ArcGIS Dashboards Questions

ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

Amanda__Huber — Thu, 22 Jul 2021 19:28:35 GMT

Hello ArcGIS Dashboard Team,

We've recently run into an issue in our ArcGIS Dashboard where our Arcade FeatureSet expression only returns 2,000 records (therefore missing a large chunk of our data). We have been unable to find documentation stating that there are record limits for arcade expressions.

Can somebody provide us more insight into this issue we're seeing?

Thanks,

Amanda Huber

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

KenBuja — Thu, 22 Jul 2021 21:14:09 GMT

The standard limit on the number of records returned on a layer in AGOL from a query is 2000. You'll have to increase that limit to get all your records, but depending on how large your dataset is, this could be a significant performance hit.

https://support.esri.com/en/technical-article/000012383

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

Amanda__Huber — Fri, 23 Jul 2021 14:10:26 GMT

Hi @KenBuja ,

Thanks so much for the quick reply and documentation.

For better understanding, how would one increase the number of records returned in Arcade not the physical service (no physical REST)? We're not seeing issues with the service's max records, only the arcade expression.

Thanks!

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

KenBuja — Fri, 23 Jul 2021 14:44:13 GMT

I'm assuming that the Arcade expression needs to make a query to the service to get the records.

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

Amanda__Huber — Mon, 26 Jul 2021 14:30:02 GMT

Hi @KenBuja,

Yes the arcade expression is querying the service, but the service is in our Enterprise Portal (not AGOL hosted- no threshold issues).

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

CMV_Erik — Wed, 13 Jul 2022 17:14:22 GMT

I am experiencing the same issue, and it doesn't seem to be with the source data. I am trying to create a dashboard to show tree assets, and want a widget to show ones with duplicate IDs (need to be corrected). I can create a variable from FeatureSetByPortalItem that returns the correct number of records (~30K), and this is reflected in my indicator widget. If I apply GroupBy to the records returned by FeatureSetByPortalItem so I can get a count by ID, it's now limited to 2000 records.

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

AlinaTarmu_SCGIS — Wed, 24 Apr 2024 18:38:41 GMT

Same issue here. Quite frustrating.

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

sophered — Tue, 23 Jul 2024 12:30:19 GMT

Same issue! seems like groupby does not work on large datasets

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

Marisa_Kordecki — Fri, 26 Jul 2024 17:47:56 GMT

Were you ever able to find a solution or workaround?

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

CMV_Erik — Fri, 26 Jul 2024 23:30:42 GMT

Not directly. I wound up preventing new duplicates in FieldMaps using Arcade to retrieve the existing IDs and preventing the workflow status field from being Editable if a duplicate ID was entered in the ID field. That solved our main problem.

If I remember correctly, I think at that point I used the ArcGIS API for Python to get a list of duplicate IDs and just worked through them. I was planning a workaround to schedule a Notebook to use the API to write the duplicate IDs to a table periodically (never real-time results, but good enough), but it looks like I never needed to go that far.

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

Marisa_Kordecki — Mon, 29 Jul 2024 11:48:29 GMT

Thanks!

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

sophered — Mon, 12 Aug 2024 12:48:51 GMT

what did that arcade expression look like? was there a particular function that you used that was helpful for retrieving those existing ids and preventing duplicates

Re: ArcGIS Dashboard- Arcade FeatureSet Expression Record Limit?

CMV_Erik — Mon, 12 Aug 2024 15:57:50 GMT

@sophered The relevant bits are below. I'm using this as one test to see if the status field should be Editable, and the user can't finish their task without changing the status field. The form reminds them the asset ID needs to be unique. While they can technically save the record with the duplicate ID, it won't move off their task list until it's unique, so the process is self-correcting.

I find FeatureSetByName very convenient to get data from other layers in my map.

var future_id_check = True var match_count = '' var this_future_id = $feature.proposed_id var lyrname_tree_assets = 'GIS Tree Data' var fs_GIS_assets = FeatureSetByName($map, lyrname_tree_assets) if (!IsEmpty(this_future_id)) { var fs_matching_assets = Filter(fs_GIS_assets, "ASSETID = '" + this_future_id + "'") match_count = Count(fs_matching_assets) if (match_count != 0) { future_id_check = False } }