topic Re: clustering 15,000+ items using javascript API and a FeatureServer or MapServer in ArcGIS JavaScript Maps SDK Questions

clustering 15,000+ items using javascript API and a FeatureServer or MapServer

SeanRyan — Fri, 16 Nov 2012 09:42:34 GMT

interesting problem:

ArcGIS server, that has a FeatureServer and a MapServer, serving layer with 15,000+ items.

problem: the service is configured to return max 1,000 items per query.

Currenty, my javascript page is using the FeatureServer, which is great, as I can use Feature Definition Expressions to filter the items.

problem: the FeatureServer only ever returns about 2,000 items, which means that when zoomed out, then whole areas of the map are empty!

note: requirement is that user should be able to view map at global scale (zoomed out) AND be able to see all 15,000 items, or at least clusters.

proposed solution:
==============
re-configure the *MapServer* to return max 8,000 or ideally 20,000 items.
change javascript, to send query (with params to filter) to retrieve ALL matching items ( up to 15,000+ items ).
then javascript will use a custom clustering solution (taking an example from the ArcGIS help site) to cluster the items, so that we can draw a reasonable number of items in the browser, say 500 items.

PROBLEM: the ArcGIS server is outside of my control, and the admins might NOT agree to increasing the max items returned from a query, which is currently limited to 1,000 items. this means it would take up to 16 queries to populate the map, which is frankly silly !

QUESTION: has anyone got an alternative solution ?
for example - could the *FeatureServer* be re-configured to support clustering ?
note: we need to work with individual items, as we have a complex filtering requirement (with about 20+ different attributes that the user can filter on) so a solution with say Country layer or State layer etc is not good.

I have tried setting the 'generalising' parameters, setMaxAllowableOffset(), but this seems to have no effect.
is setMaxAllowableOffset() obsolete, since it is for 'generalising' which is supposed to be done automatically by Feature Server ?

Have also tried MODE_SNAPSHOT but found that MODE_ONDEMAND actually brings back more items, and so, is more useful.

Possible solution: just write my own web service to serve up the map points.
however, this is a *lot* of work, and there could be deployment issues, as the Production Environment is very restricted.

Any help or tips would be most appreciated ....

Re: clustering 15,000+ items using javascript API and a FeatureServer or MapServer

HinnerkGildhoff — Wed, 09 Dec 2015 11:11:04 GMT

The spatial engine of SAP HANA offers clustering at database level. So you can cluster big amounts of data in the backend (in-memory) and transfer just the cluster result via features services to the client. If this sounds interesting for you, here's a link to the official documentation:

Spatial Clustering - SAP HANA Spatial Reference - SAP Library

Re: clustering 15,000+ items using javascript API and a FeatureServer or MapServer

ReneRubalcava — Wed, 09 Dec 2015 14:39:19 GMT

There are a couple of client-side solutions.

Esri/cluster-layer-js · GitHub

With this one, if there are 15,000 records, but max limit of 1000 per request, it will make 15 requests and cache them. Similar to a SNAPSHOT mode of FeatureLayer. Performance may vary, but once data is downloaded, draws should be fast.

There's this FlareClusterLayer

nickcam/FlareClusterLayer · GitHub

But I don't know if it can pull down any more data than a regular FeatureLayer.

Re: clustering 15,000+ items using javascript API and a FeatureServer or MapServer

HinnerkGildhoff — Wed, 09 Dec 2015 16:28:38 GMT

Clustering on the client works fine with javascript if you have a couple of thousands points.

For millions, I would do it in the backend