Dynamic Point Clustering for data of 100,000+ size

[StuartThomson1]

Hi guys,

I’ve been trying to integrate ArcGIS into our application. The basic requirement is to draw points, and allow selection/querying of them with drawing tools. It will be required to be able to show 100,000+ points. When I draw this many points, it’s taking 15 seconds to redraw when you zoom in or out which is obviously unusable.

I have seen a few point clustering examples which seems the best solution, but they don’t seem to work how I want. I’ve already got all the data I need accessible before I create the map (lat/longs, coordinate system), so I don’t need to run a query, I just need to draw the points.

It would be ideal if there was a numerical limit to the numbers of points on the screen drawn at one time and when you zoomed in further, if the points could be displayed then they would be. What’s the best way to approach clustering, including being able to select the points with drawing tools, and how much of this do I need to write myself?

[DallasShearer]

I'm assuming you are drawing graphics and not using a published service to render the image. Ideally you would want to use a published service and let a beefy server do the work. But this may not be possible if you have chosen to use graphics. Take a look at this link it is the ideal way of point clustering using graphics.

Point clustering | ArcGIS API for JavaScript

[StuartThomson1]

Thanks - I am drawing with graphics. The data comes from our own bespoke server running in Tomcat in a JSON format. I initially tried drawing the points with graphics but that doesn't scale very well. Our Java implementation is able to draw 100k points just fine.

When you mean a published service, you mean getting a server to actually create the image and then just feeding that back to the browser and displaying it? Wouldn't that complicate selecting of points and GUI interactions due to it being only an image, and how would i go about doing that?

[DallasShearer]

You can use this API without having ArcGIS Server installed, if this is your case where you are using the API without an implementation of ArcGIS Server; then you won't be able to publish a map service. If you do have ArcGIS Server installed then you can add and display your data in an ArcMap document, publish it to your ArcGIS Server and then consume the published service in your web application. Here is a tutorial link for doing that.

ArcGIS Help 10.1

Also, you may be able to publish the data to an ArGIS Online account and consume it from there, but I am less familiar with this process.

Nevertheless, if you are unfamiliar with this process you may find it less troublesome to achieve your goal using the link I posted in my 1st reply.

[StuartThomson1]

Thanks for another helpful reply!

The points I'm displaying are not going to always be the same. They will be created dynamically from a search depending on what parameters the user selected right before the map is created. At the moment, they are coming into the browser as JSON. I've looked at the clustering example you sent as well as the code behind it and it seems to do most of what we want, and the code is actually based on reading a JSON file containing all the points. I'm unsure if selecting the points with drawing tools (squares, polygons, etc) would select all the points behind the clusters correctly. Would that still work?

The second option of creating a server sounds interesting. Does this create actual rendered images on the server end with the points flattened with the map imagery available as a tiled layer? That obviously would be very fast on the front end when you pan and zoom but I'm guessing that the functionality would be reduced, ie, you can't click or interact with the points? I'm not sure how this would work.

Say a user has put in a bunch of search parameters in our program, the server will then respond with a large JSON response containing all the points to be plotted. Our back end is Java based, so presumably I could create a new ArcDocument for each request in the back end and then consume that with the web api?

[DallasShearer]

The ArcGIS server is an application installed on a server. The initial license fee is typically 25k.

even thought the rendered image is in fact an image you can use draw tools to interact with the displayed features. What happens is it takes the geometry of the drawn object and performs a spatial query against the dataset on the server. The results are return and then you do whatever with the results... like draw graphics on the screen to show which features were selected or display tabular results in a table.  To answer your fist question you would want to do something similar like get the geometry of the drawn feature and use that as parameters to filter out the originally returned data.

You could possibly create your own application to generate the image on the server but you would need to to extensive test make sure it lines up correctly. If you have the 25k to put towards the project then purchasing the ArcGIS server license would prove worth it. Also, you may want to look into getting an EDN license for ArcGIS server so you can test this out before you pay the big buck on ArcGIS Server.

[DallasShearer]

also your JAVA implementation would be doing the work on the server side, versus this JavaScript implementation that would be doing the work on the client side and the browser being used would make a difference in performance as well.

[ReneRubalcava]

If you are able to publish the data to ArcGIS Server or ArcGIS Online, I extended the original clustering example to work more like a FeatureLayer via URL, which includes caching and working around the default 1000 feature limit. I put a post up on it here. Let's talk clusters.

[StuartThomson1]

Thanks for that! I left a comment there asking 3 questions