topic Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API in ArcGIS JavaScript Maps SDK Questions

Guidance Needed on Rendering Large Datasets with ArcGIS JS API

FactSetResearchSystems — Thu, 24 Apr 2025 05:05:42 GMT

I'm working with @ArcGIS/core@4.21.2 in a Vue application and facing some performance challenges when trying to display a large dataset on a map.

We’ve uploaded a CSV file with around 15 million rows into a hosted feature layer. When applying a filter, the resulting dataset includes approximately 350,000 points. However, the data is being fetched in batches, and seems to be limited by a maxRecordCount setting. Even though I attempted to update this setting using the ArcGIS updateDefinition API (as described here: https://developers.arcgis.com/rest/services-reference/online/update-definition-feature-service-.htm), the value still shows as 1000, and the actual requests seem capped at 4000 records.

In some cases, even fetching 4000 records takes up to 2 minutes, possibly due to how offset-based pagination is handled.

Given this situation:

What is the best approach for efficiently displaying such large datasets on the map?
Would it be better to use techniques like tiling, generalization, or server-side clustering?

Any recommendations or best practices would be greatly appreciated.

Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API

mgeorge — Thu, 24 Apr 2025 22:07:32 GMT

Hi @FactSetResearchSystems, are you using a FeatureLayer? We should be downloading pages of 8k features. I'd make sure caching is enabled. Here's an example of a map loading about 350k features: https://codepen.io/matt9222/pen/MYYpPRK?editors=1000

Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API

FactSetResearchSystems — Fri, 25 Apr 2025 04:59:45 GMT

Yes I am using the FeatureLayer itself, this is interesting, you maps loads pretty fast, where as mine loads very slow, may be its the query to fetch the 350k records out from 15million which is taking time. How did you achieve 8k , my feature layer doesn't always give more than 4k. Is there any featurelayer or server setting you have done .

Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API

BBarbs — Fri, 25 Apr 2025 13:59:52 GMT

We also have a few Vue apps that consume data from large datasets (90 million records). We wound up creating an API endpoint connected to our database that the app sends queries to, that way only what the client needs is sent to the app itself (in GeoJSON format). This usually winds up only being a few thousand points with a max of about 15k though.

I'm not sure if there's an easier way, but it's worked pretty well for us so far.

Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API

mgeorge — Fri, 25 Apr 2025 16:42:21 GMT

Hmm strange -- it should be the default for AGOL hosted. Is there any kind of proxy that your service is running through? Also do you see if things are getting cached? Might be slow the first time, but I'd expect it to speed up once things are cached (there's a feature tile cache on the server and a CDN cache).

Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API

mgeorge — Fri, 25 Apr 2025 16:43:24 GMT

That's a valid option -- only downside is that GeoJSON is not a very efficient format, though should be fine for a few thousand points 👍

Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API

FactSetResearchSystems — Wed, 30 Apr 2025 05:56:41 GMT

I see my default its cached for 30 mints and i have increased it to 1 hour. I think its getting cached on the client side since i see the subsequent requests loads much faster.

Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API

EricLake1 — Wed, 16 Jul 2025 18:38:46 GMT

What do you mean by "make sure caching is enabled" ? Is this the CDN cache that makes your sample download 350k records so fast? For us, that kind of data is very slow on our servers.

Re: Guidance Needed on Rendering Large Datasets with ArcGIS JS API

mgeorge — Wed, 16 Jul 2025 18:51:18 GMT

@EricLake1 yes this is using AGOL hosted here, though the cold cache times shouldn't be that bad for this either (you can test this yourself by adding a definition expression like 4=4 to the layer). Also note that if you change the initial map zoom for this to be something like 12, the map will instead use tile queries.

Is your backend actually bandwidth limited or are you seeing slow db timings? If it's the latter, deploying a cache in front of your service is probably the way to go.