Publishing large vector data with greater than 1 million records

[BrettShaver]

I am trying to publish a vector dataset (points) with greater than 1 million records and many fields.  Ultimately it will be used solely in a query widget in a WebApp.  What is the best method?  ArcGIS server, hosted feature layer in AGOL, etc.?  I've tried both thus far and both are pretty slow and clunky.

[DanPatterson_Retired]

have you considered parsing into smaller datasets either spatially or by attribute?

[BrettShaver]

Hi Dan - Unfortunately it can't be.  The whole benefit of the query widget for this dataset is it enables users to query any data.  The geographic extent of the dataset is very small, but it's a large number of records over 30 years

[KellyGerrow]

HI Brett,

Are you able to share the data? If its private, you can invite my username (kgerrow_cloudygis) to a group if you don't want to share the data publicly.

Can you provide a description of the data? What is the data of? How will it be queried? How many results are being returned?

Additionally as a wild suggestion, not knowing much about your specific use case, you can rebuild your index which can result in performance improvements depending on what is causing the slowness: ArcGIS REST API 

-Kelly

[BrettShaver]

Hi Kelly - thanks for the response.  Unfortunately I can't share any of the data since the project is highly confidential.  

However, the data is a chemical dataset for groundwater sampling events for thousands of locations for over 30 years.  The query widget will allow the user to query certain compounds, values, dates, and other fields from the data.  There's approximately 1.1 million records and 20+ fields.  The data is not symbolized, or even turned on.  The only thing it is used for is an input to the query widget. 

Currently, I have the data published to AGOL as a hosted feature layer (which I thought would be faster than hosting it to our own server through AGS).  It seems to be slowing down the web app somewhat significantly, and often times it will take 3-4 minutes for the web application to load while waiting for this dataset to respond.

Any help would be greatly appreciated!

Thanks,

Brett 

[KellyGerrow]

HI Brett,

Thanks for the additional details,

What is the Max record count that you have set up? Do you want to view all of the points drawn in one panel? If so, I'd suggest creating a tile layer for visualization and use the feature service to query.

Here are some other resources that may be helpful:

Best Practices for using Tile Layers as Operational Layers | ArcGIS Blog 

Q&A questions about Layers and Layer Types:

http://www.esri.com/about/events/uc/get-involved/q-and-a#category=Components of ArcGIS - ArcGIS Onli...

Is there a limit of 1,000 features that can be displayed in Map Viewer and the Web Apps hosted in ArcGIS Online?

Should I increase the max record count on hosted feature services so more than 2000 features can display in web maps and apps?

What type of layer should I use in my web maps and apps?

-Kelly

[BrettShaver]

Hi Kelly - The layer with the 1.1 million datapoints is actually not even visualized outside of the whatever results the query widget spits out.  We've disabled the option to view it in the map, or to see it in data layers.  Because we aren't visualizing it, and only need the attributes to be able to feed into a query widget, I don't think tiling will be helpful.  

Re: question on max record count - When publishing the service originally from ArcMap as a hosted feature layer, it was set to have the server return 1,000,000 records. 

[ThomasColson]

Are you able to use native Leaflet or another Java viewer besides web app builder? I've found WAB to be a little slow in handling large requests, such as yours. Then feed GeoJSON from your AGOL feature service into Leaflet directly. The downside, of course, is that you have to code the query and return the results, which WAB will do for you.....

[BrettShaver]

Thanks Thomas, but no.  Switching the platform out of AGOL will cause more headaches and the project doesn't have the budget for it.  Benefits of AGOL web app builder is we can design and push these products quickly, and edit quickly.  

[ThomasColson]

You stuff would still be on AGOL. It's been a while, but I vaguely recall ESRI publishing some leaflet examples....maybe on their GitHUb site? And...I'd have to log this thru wireshark or fiddler...but it may be that WAB is pulling the entire feature service and the query is happening in the browser...are you able to tweak your query widget so that the query is being passed as an API call to AGOL, which, in theory, should be faster? Are you able to reproduce your query string as a URL against the rest service, while hitting "F12" in Chrome, and look at the "Performance" tab, do you see the same latency there?