I need to visualize 100k records

[DavidElies]

Hi all!

I have over a million species occurrence points, and users are supposed to be able to select points by species.  Problem is our most common species show well over 100k points.  In the past I've created an arbitrary grid where each cell has the total number of points within that cell.  Then I just show the grid cells that contain at least 1 point.  Queries weren't allowed until the user zoomed to a specific level, where the individual points were limited to a few thousand.  This time I can't limit queries to certain zoom level, nor can I afford to use thousands of separate grids with the point counts for each individual species.  I would love to use some form of heat map (my grids were a rudimentary sort of heat map), or something else to summarize the points when zoomed way out.

I was thinking of using the grid to query all the matching features in each cell, but I didn't know if I could send 150 or so queries at the same time.

Has anyone done something like this before?  I could use any guidance you can provide.  I'm using ArcGIS Server 9.3.

Thanks for your help!

[BenHolmes]

Hi David

Not really JS API, but you might want to look at density mapping. You would need the spatial analyst extension for it. You can do it at desktop and then load the raster into a map service.

Other than that, if you want to look at the points, you may want to look at multipoints rather than single point data. Again not server, but in the data.

Hope that gives you some ideas!
Ben

[DavidElies]

Thanks Ben,

I am really trying to avoid having to make separate data for each species.  This would be thousands of separate tables or shapefiles.  I've had some luck with actually sending the 150 or so queries and getting the results.  Since they're asynchronous, the points just keep appearing as they become available.  I don't think this is the best way to do this, but it does return most of the points.  If I am going to continue to do this, I need to know how to know which async response matches which request.  Is there anyway to do that?  Do you have any other ideas of how to show the density of points without having to pre-create hundreds of density maps?

[StephenLead]

David,

You could look at clustering the points - the map here shows an example of this.

I used the clustering code from the Resources page, and fed it an array of all the points, which it then returns as clustered points.

I've tested with around 10k points and it works fine - you may see a slowdown with 100k and 1M points.

Steve

[DavidElies]

Thanks stevel,

I think that would be a great way to go.  I don't think I was clear enough on what my problem actually was.  I can't return 10s of thousands or 100s of thousands of records from the server, so I don't have all of the records in order to cluster them.  I will definitely look into clustering once I figure out how to get all the points.  Thanks again!