GeoDev Webinar: Building Web Apps That Visualize Large Datasets

[AmyNiessen]

At the end of May, we hosted a GeoDev Webinar on one of the latest blog posts the came from Kristian Ekenes, who works on the Senior Product Engineer on the ArcGIS API for JavaScript team. He wrote a blog post on Mapping Large Datasets on the Web, and since we thought this would be a great topic to cover more in-depth, we decided to host the same topic as a webinar where attendees could ask questions. There were a lot of good questions that came in but were not addressed during the live Q&A portion of the webinar, so Kristian addresses them below.

Q: What parameters need to be set to enable dynamic tile service?

A: You don't need to do anything to enable dynamic feature tiles. You get them out of the box with online hosted Feature Services and Enterprise feature services. See attached matrix for more specific information on versioning.

Q: Can you show me where the quantization parameter is defined and for which type of services?

A: Quantization parameters are query parameters for the feature service. You can directly query data in quantized format using the JS API's Query object. See the doc here:  https://developers.arcgis.com/javascript/latest/api-reference/esri-tasks-support-Query.html#quantiza.... But you don't need to worry about that. The JS API takes care of querying the data in quantized form for you.

Q:  I have an ArcGIS Online license and I uploaded a 4GB data set as a CSV that created a table feature layer, that then is used to create a map. The thing is that I must update this data set every week. Is there some way, or an architecture that let me automate this? I already updated the data set with ArcGIS REST API, but it doesn't reflect on the maps or in the table feature layer.

A: Yes. You can automatically apply edits to this feature service using the ArcGIS Python API. Though this isn't my area of expertise. I would reach out to someone on GeoNet in a Python discussion for a more specific answer. The thing to remember though, is that once you update the data, the old feature tiles are automatically replaced with tiles containing the new/updated data once a new query for that data is made. So you don't have to worry about the backend taking care of that for you! 

Q: I am using one feature service to different maps. How can I filter data based on a map?

A: You will want to contact Esri Technical Support for this.

Q: When publishing a feature service to ArcGIS Online, will this eliminate the restriction on how many features you can render?

A: There isn't a limit to the number of features you can publish. The limitation you may encounter is with storage. The number of features you can render is dependent on the client loading the data, the network speed/latency, number of attributes required, etc.

Q: Will example code be available in GitHub?

A: Example code is here: https://github.com/ekenes/conferences/tree/master/ds-2020/large-data and here: https://github.com/ekenes/conferences/tree/master/ds-2020/plenary.

Q: Do we have to set the scale in the feature layer definition?

A: One way to avoid loading too many features unnecessarily is by progressively filtering out data based on view scale. This isn't the only way, but just one method without having to load the layer multiple times.

Q: I am not familiar with CDN cache. How can I optimize the performance by using this cache?

A: You don't need to do anything to take advantage of this. It just applies to public feature services. If you have a public feature service hosted on ArcGIS Online, then you automatically benefit from the CDN cache.

Q: Is there a dataset where we can get access to US population or zip code density? I could not seem to find any.

A: The Living Atlas of the World has zip code layers you can freely use as well as up to date population estimates through the ACS. I highly recommend searching there. https://livingatlas.arcgis.com/en/browse/#d=2&q=zip%20code

Q: Why are you cloning the renderer for Feature Layer?

A: I'm cloning the renderer so when I reset the renderer the layer will detect changes and re-render the data. We don't watch all renderer and symbol properties for changes for performance reasons. Therefore you must clone the renderer, make your modifications, then set it back on the layer. This is your way of deliberately telling the layer  a change has been made and it needs to redraw the features.

Q: Can you apply this visualization techniques in ArcMap or ArcGIS Pro?

A: Not all of these techniques can be applied in our desktop software. You can set the same renderer types…graduated symbols, color visual variables, etc. But you cannot update the renderer based on another attribute like time or depth. The time and depth UI sliders in Pro and ArcMap perform filters of the data. I'm rendering all features in the JS API and updating the renderer rather than performing a data filter. This allows me to avoid loading duplicate geometries just to show different data values. So no, you can't use all of these techniques in ArcGIS Pro/ArcMap. Also, you cannot set up the size range by scale in ArcGIS Pro.

Q: What about applying this visualization technique in ArcMap or ArcPro regarding line color thickness?

A: Regarding the scale-dependent line thickness in the pipes example. You can configure that in Pro, but it's a different approach than in the JS API. In Pro, you set a reference scale and a size that will render the lines at that scale with a specific size. When you zoom in or out, the line width will adjust linearly based on the difference between the map scale and the reference scale. You can set more stops in the JS API to do it.

Q: Can the Arcade expressions and other things be leveraged in a context wherein everything is, by intent or design, is cached on the client in memory or with the CDN -- specifically avoiding everything except the initial call to the originating ArcGIS service as a REST service? 

A: Arcade can execute against client-side features and you can query your data client-side, thus avoiding another round trip to the server. You first have to ensure that you actually have all the data available on the client though.

Q: Could you provide the code for this examples please?

A: Example code is here: https://github.com/ekenes/conferences/tree/master/ds-2020/large-data and here: https://github.com/ekenes/conferences/tree/master/ds-2020/plenary

Q: What difference between filter scale on API and display scale on mxd then publish to feature service?

A: Hopefully I get this right…display scale in ArcGIS Pro/ArcMap is similar to visibility scale in the JS API (layer.minScale/layer.maxScale). The difference is the visibility scale determines when a layer will be queried and displayed based on map/view scale. The filtering by scale still queries the data regardless if there is visibility scale (if there is one the visibility scale is still honored). You're just being more deliberate about filtering out data, such as smaller or less meaningful features that aren't needed for that scale. So you still see data, just not all the features in the approach where you filter based on scale.

Q: How do you normally explain clustering to the lay person?

A: Clustering is a method of reducing the number of features in view by aggregating features into clusters based on a predefined cluster radius. Larger cluster graphics indicate areas that have a higher density of features. Smaller cluster graphics indicate areas with fewer features.

Q: Is clustering available in Portal as well as ArcGIS Online?

A: Yes

Q: Could you provide the codes and the links for this examples please?

A: Example code is here: https://github.com/ekenes/conferences/tree/master/ds-2020/large-data and here: https://github.com/ekenes/conferences/tree/master/ds-2020/plenary

Q: How do we deal with the time based data? For example, I am dealing with vehicle speed data which is 70k line per minute, and I want to show animation that will last for one hour.

A: I'm not sure I understand the case. Animations can be tricky…perhaps a question to post on the ArcGIS API for JavaScript GeoNet community with more specific details?

Q: If you need to update the layer, how do you update it?

A: Just apply edits or set the properties. 

Q: Do you offer tailor made training on web app development? I find the One Ocean app cool and would like to develop one for my region.

A: No. But I regularly contribute to the ArcGIS blog where I discuss details on some of these projects like One Ocean. You can read it here: https://www.esri.com/arcgis-blog/products/js-api-arcgis/mapping/mapping-large-datasets-on-the-web/ other JS API blogs can be searched on this page: https://www.esri.com/arcgis-blog/?s=#&products=js-api-arcgis.

Q: Thank you for the examples of using queries with a Feature Tile Cache. Can you also use  Filter Widget, or reporting tool widgets with a Feature Tile Cache?

A: Any time you filter your layer, the data is requested in tile format, which means it is automatically cached for you. So you don't have to worry about configuring it. As long as the JS API recognizes the query as a repeatable one, you leverage the feature tile cache.

Q: Can you use this techniques with rasters or grids?

A: Not at the moment. This only applies to vector data.

Q: Is there a GitHub link for the EugeneTrees - Cluster example?

A: Yes, you can find it here: Map Viewer 

Q: Is it best to limit the fields (attributes needed) in ArcMap or ArcGIS Pro before you publish or elsewhere (i.e. in the web map configuration)?

A: Not necessarily. You can also limit the fields using a hosted Feature Layer View. You can also limit them in the outFields of the layer in the JS app. If you have a long list of fields though, the query won't be cacheable (query stings must be less than 2048 characters), so it is best to limit fields in those situations whether it is from Pro or a hosted layer view.

Q: Some of the features you presented (such as adjusting the size of a line by scale, or definition queries by scale) look really interesting, but I am developing web apps in web app builder. Can you use those tools in a GUI driven environment? Or would it have to be within the code?

A: When you style a layer using the new Map Viewer Beta, you already take advantage of the scale-driven symbology by default. But the renderer must be authored in the viewer. That means resetting it there even if you have one saved to the layer. Or you can simply load it in the new map viewer beta and check the box. Read this blog for more information - https://www.esri.com/arcgis-blog/products/arcgis-online/mapping/auto-size-by-scale-now-available-in-... Regarding the scale-driven definition queries, you have to do that in code. There is no GUI for it.

Q: Hello, how can I access the JavaScript backend code which you have reviewed?

A: Example code is here: https://github.com/ekenes/conferences/tree/master/ds-2020/large-data and here: https://github.com/ekenes/conferences/tree/master/ds-2020/plenary

Q: How much can you improve performance of a dataset by adjusting text lengths of an attribute table? Do longer lengths greatly reduce draw speed in a feature service?

A: Longer lengths will reduce speed. But it may only matter if you have a lot of features and/or a lot of fields you are loading. We're continually improving draw times though, so it may matter less and less. You should see a significant improvement here later this year.

Q: Is there any documentation on geometry thinning?

A: You can read more about it in this blog - https://www.esri.com/arcgis-blog/products/js-api-arcgis/mapping/mapping-large-datasets-on-the-web/#g... - But you can also read about it in Pro documentation under Select Layer By Attribute. Fundamentally, what I mean by "geometry thinning" is filtering out unnecessary features based on their geometry...whether they are inside or outside an area of interest...or in this case...whether there are too many points in a grid (e.g. stacked on top of one another, or even grid resolution).

Q: How do I cluster the thousands of POIs from different categories for better performance?

A: To cluster by category, you need to set up different layers with definitionExpressions based on each category…I'm not sure if you get better performance though. You should get decent performance in clustering with a few thousand features (even in the hundreds of thousands). But if you have way more than that, then you'll need to enable clustering on your service, which isn't fully supported in the JS APi yet. Though it's coming soon...

Q: When will snapping will be available in 4.x?

A: It is planned, but there is no specific date set.

Q: If I set scale range in mxd for a feature, but I want to see all data in attribute table, I receive slow results. Why?

A: This doesn't appear related to the JS API. I would contact Esri Technical Support. 

Q: Would you happen to have any advice for improving dataset performance within an ArcGIS Dashboards?

A: I would ask that question on GeoNet in the ArcGIS Dashboards discussion. You'll get someone who works on that product that will provide you with a better answer than I can give.

Q: How frequently is the data is cached? Can we change the frequency?

A: You can change the frequency using the maxAge parameter in the layer's settings in ArcGIS Online. Go to the layer item. Click the "Settings" tab. Scroll down to "Cache control". There you can control how long clients will have to wait before seeing an updated cache. That applies to editable layers where the features/attributes may change. Once the tiles are cached, they will stay that way until an edit is made.

Q: Are there any plans to move Web AppBuilder for Developers to work with the 4x API?

A: Yes. You'll need to contact the Web AppBuilder team though. You can reach them on GeoNet.

Q: How do I publish feature tile services instead of feature services? And it sounds like feature tile service is better than feature service. Should I use feature tile service all the time? What's the advantage of feature service that feature tile service doesn't have?

A: Feature services automatically query data as dynamic feature tiles. You don't have to do anything to take advantage of this functionality. It's all happening behind the scenes for you.

You can find a recording of this video on our GeoDev Webinar playlist on YouTube. If you would like to download the slides to the webinar, you can do so here: https://github.com/ekenes/conferences/raw/master/ds-2020/large-data/geodev-slides.pptx. 

We hope you enjoyed this installment of the GeoDev Webinars! You can find all of our GeoDev Webinars on go.esri.com/geodev. Until next time...