Large amount of data points to esri-leaflet

NickySonnemans · ‎11-23-2022

Hi,

I have made a leaflet web application (https://alsi.gie.eu/data-visualisation/maps), next I wish to add layers on top which are hosted on https://services.arcgis.com/W1S2EV01pfLNmn8R/arcgis/rest/services/network1908/FeatureServer.

I have tried using the layers directly hosted on the featureserver, but found this too be very data intensive. Whenever I zoomed in, the entire application crashed; because of the redrawing. I have added it as such:

L.esri.featureLayer({
url: 'https://services.arcgis.com/W1S2EV01pfLNmn8R/arcgis/rest/services/network1908/FeatureServer/9',
style: function () {
return { color: "#c29ed7", weight: 0.5, opacity: 0.5, stroke: true, fill: false };
}
}).addTo(map);

I then came across similar issues on https://github.com/Esri/esri-leaflet/issues/738. I figured that I would need to either convert the FeatureServer to MapServer or find another solution that basically projects an image instead of the entire shapefile onto the leaflet map. Can anybody point me in the right direction here? It would be really helpful if you explain the steps, as I'm not very proficient in ARCGIS/ESRI.

Raul_Jimenez · ‎11-23-2022

Hi @NickySonnemans ,

First things that comes to my mind:

You can use a cluster point renderer
If you own that layer you can:
- Publish the feature layer as a vector tile layer and load it using the vectorTileLayer plugin
- Publish the feature layer as a raster tile layer and load it using as a tiledMapLayer

Note: if you publish the layer as a tile layer you will need to write a mechanism to retrieve the attributes from the features from the service (which will be a little bit tedious).

I hope you don't mind, but I'm going to give you another advice... 😅.

I would recommend you to take a look to the JavaScript API. For an application like this one the learning curve shouldn't be big, and here you can see who it can load the ~15.000 geometries that this service has easily here: https://jsbin.com/lonuhop/1/edit?html,output

Because you said you are not very proficient in ArcGIS, I highly recommend you to take a look to the Map Viewer (learning path), it is an awesome tool (accesible for free) that combined with the ArcGIS API for JavaScript will help you to speed up your development process, helping you to configure nice symbols and popups really quick: you can start from here. After saving your map an ID is returned in the URL. You can grab it and use it to load the configuration you made with a few lines of code. After doing that, you can access the configuration object and adjust, modify, and customize the behaviour of the rest of the application programmatically (as I was showing here).

I hope this helps,
Raul

DanielDormont · ‎11-30-2022

Hi @Raul_Jimenez I've been wondering something similar myself. In my application we use Leaflet just because it's a bit more generic and we were already familiar with it before we began integrating with Esri, but we have been running into similar performance issues displaying complex Feature Layers. What is the limitation that is preventing these from being displayed efficiently in Leaflet when the ArcGis Javascript seems to handle them ok?

Thanks, Dan

Raul_Jimenez · ‎12-09-2022

Hi @DanielDormont ,

There are multiple optimizations been done behind the scenes. @JeremyBartley explained some of them on Behind the scenes, What the JS API and ArcGIS do for you.

But in summary ArcGIS API for JS use WebGL (Leaflet doesn't) plus the ArcGIS API for JS make use of all the optimizations included in the Feature Service to reduce the data transferred and produce faster responses (tile caching, PBFs encoded responses, geometry quantization, etc.) <- and as far as I know, the Esri Leaflet FeatureLayer.js doesn't as can be seen here:

There you can see how Esri Leaflet request a GeoJSON from the service, plus every time you zoom in & zoom out send a bunch of requests to retrieve only the geometries needed (but do not cache, compress or simplify them) <- it would be great if someone in the community would open a PR to help improve it (but I think there are pretty complex optimizations in the JSAPI that wouldn't be easy to replicate).

In my application we use Leaflet just because it's a bit more generic

Interesting, what do you mean by more generic? Is there anything we could ask the JSAPI team to do to make it more generic?

we were already familiar with it before we began integrating with Esri, but we have been running into similar performance issues displaying complex Feature Layers

That's a common situation. Let me know if there is anything else we can do to support you or make your life easier learning how to integrate with ArcGIS.

Cheers,
Raul

DanielDormont · ‎12-11-2022

Hi @Raul_Jimenez thanks for the reply, that's very interesting. I guess by "generic" I just meant that Leaflet can pull data including basemaps from any source; we have some customers who have their own custom data sources they want to use. In fact, we even have a few applications where the coordinate data aren't geographic at all, and we use https://leafletjs.com/examples/crs-simple/crs-simple.html to display them.

And leaflet does work with WebGL, via esri-leaflet-vector, doesn't it? In fact that's something I've been wondering. Since we already use esri-leaflet-vector to show vector basemaps, is there not a way to use that same technology with feature layers?

Raul_Jimenez · ‎12-30-2022

Sorry @DanielDormont for the delay!

And leaflet does work with WebGL, via esri-leaflet-vector, doesn't it? In fact that's something I've been wondering. Since we already use esri-leaflet-vector to show vector basemaps, is there not a way to use that same technology with feature layers?

I don't think there is an easy way.

As you can see vector basemaps relies on maplibre-gl-js, and basemap tile services and feature layers services are very different. esri-leaflet-vector, but it would be better to ask in the repo.

I guess by "generic" I just meant that Leaflet can pull data including basemaps from any source; we have some customers who have their own custom data sources they want to use.

🤔... did you know that the JS API can also be used with any custom data source? I mean, there are some standard layers (GeoJSON, CSV, KML, WFS, ...) , and if none of them fit, you can also do it by drawing directly in a GraphicsLayer (in memory) or populating a FeatureLayer using an array graphics (example).

In fact, we even have a few applications where the coordinate data aren't geographic at all, and we use https://leafletjs.com/examples/crs-simple/crs-simple.html to display them.

I agree that for use cases like this one I wouldn't use the ArcGIS Maps SDK for JavaScript either :).

I hope you have a happy new year eve!! 🥳

Raul_Jimenez · ‎02-10-2023

Hi @DanielDormont I just read "Evaluating the Performance of Three Popular Web Mapping Libraries: A Case Study Using Argentina’s Li..." (compares Leaflet, Mapbox GL JS and OpenLayers) and remembered me this thread:

...And leaflet does work with WebGL, via esri-leaflet-vector, doesn't it? In fact that's something I've been wondering. Since we already use esri-leaflet-vector to show vector basemaps, is there not a way to use that same technology with feature layers?

Spoiler of the conclusions 😜:

I will see if I can do my own research and include the ArcGIS API for JS too to the equation 😉, it would be nice to have a rigorous analysis considering them all and different visualization types.