nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Publishing large vector data with greater than 1 million records

BrettShaver

I am trying to publish a vector dataset (points) with greater than 1 million records and many fields. Ultimately it will be used solely in a query widget in a WebApp. What is the best method? ArcGIS server, hosted feature layer in AGOL, etc.? I've tried both thus far and both are pretty slow and clunky.

Find more posts tagged with

Accepted answers

All comments

DanPatterson_Retired

have you considered parsing into smaller datasets either spatially or by attribute?

BrettShaver

Hi Dan - Unfortunately it can't be. The whole benefit of the query widget for this dataset is it enables users to query any data. The geographic extent of the dataset is very small, but it's a large number of records over 30 years

KellyGerrow

HI Brett,

Are you able to share the data? If its private, you can invite my username (kgerrow_cloudygis) to a group if you don't want to share the data publicly.

Can you provide a description of the data? What is the data of? How will it be queried? How many results are being returned?

Additionally as a wild suggestion, not knowing much about your specific use case, you can rebuild your index which can result in performance improvements depending on what is causing the slowness: ArcGIS REST API

-Kelly

BrettShaver

Hi Kelly - thanks for the response. Unfortunately I can't share any of the data since the project is highly confidential.

However, the data is a chemical dataset for groundwater sampling events for thousands of locations for over 30 years. The query widget will allow the user to query certain compounds, values, dates, and other fields from the data. There's approximately 1.1 million records and 20+ fields. The data is not symbolized, or even turned on. The only thing it is used for is an input to the query widget.

Currently, I have the data published to AGOL as a hosted feature layer (which I thought would be faster than hosting it to our own server through AGS). It seems to be slowing down the web app somewhat significantly, and often times it will take 3-4 minutes for the web application to load while waiting for this dataset to respond.

Any help would be greatly appreciated!

Thanks,

Brett

ThomasColson

Are you able to use native Leaflet or another Java viewer besides web app builder? I've found WAB to be a little slow in handling large requests, such as yours. Then feed GeoJSON from your AGOL feature service into Leaflet directly. The downside, of course, is that you have to code the query and return the results, which WAB will do for you.....

BrettShaver

Thanks Thomas, but no. Switching the platform out of AGOL will cause more headaches and the project doesn't have the budget for it. Benefits of AGOL web app builder is we can design and push these products quickly, and edit quickly.

ThomasColson

You stuff would still be on AGOL. It's been a while, but I vaguely recall ESRI publishing some leaflet examples....maybe on their GitHUb site? And...I'd have to log this thru wireshark or fiddler...but it may be that WAB is pulling the entire feature service and the query is happening in the browser...are you able to tweak your query widget so that the query is being passed as an API call to AGOL, which, in theory, should be faster? Are you able to reproduce your query string as a URL against the rest service, while hitting "F12" in Chrome, and look at the "Performance" tab, do you see the same latency there?

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.