Design Question - Showing Dynamic Points on a Map

[ChrisSmith7]

Hello,

I have an application where dynamic points are pulled from a SQL db via an Ajax call - the points cannot live as a static map service on the server, and the SQL db from which I query is not integrated, and cannot be made to integrate, with ArcGIS Server, at least at the current moment.

The basic set-up works well - I can asynchronously retrieve the x,y points and make them available for display via a toggle button. I am creating a feature layer using a JSON feature collection/layer definition and applying a unique value renderer.

The problem is with the volume of data - I can only display a sampling of the available features... there are too many points to display at once. What I would really like to do is apply some sort of on-demand filter. The cap would remain the same; however, as the scale becomes smaller, the density of points would increase, and previously hidden points (at the larger scale) would become available.

I know we can do this for map services on the server, but what about JSON-created layers? Ideally, every time a user loads the points, a new db call would be made to retrieve fresh data as they have a high rate of change. I could reduce the refresh to a daily update if there is a way to update a static map service through an automated batch job. The complete scope of data we are talking about is tens of millions of points, although we would never show all at any scale, and would need a connection to, at the most, several tens of thousands to hundreds of thousands of points.

Any ideas would be appreciated!

[thejuskambi]

Not sure if this would be helpful. Have you tried to create a spatial view for the data and then access it via a map service. you can use the layer definition to achieve the filtering you require.

Usually, if somebody is talking about huge quantity of data, I would start thinking bigdata as an option, but then again, the volume and efforts has to be balanced. Any RDBMS now a days can handle hundreds of thousands of data. But, DB doesn't seems to be an issue for you.

I am wondering why or what use would it be to display so much of points all at the same time. wouldn't it just clutter the map. cant you filter it or convert it to heatmap or something?

[ChrisSmith7]

I actually do show the points as a polygonal heatmap - the points are needed so that users can click and obtain additional information, if they desired, and to see a general density pattern at an abstract view. Down the road, the idea would be to allow the user to filter on certain spatial, categorical, and quantitative criteria, say, within a radius of a certain lat/long, etc.

I don't expect to show all points - just a representative sample at any give moment - say an absolute max of 10k points, but for the moment, I would like to have the ability to show all points once they zoom-in and pan. In other words, the map will only ever show an absolute max of 10k points, but, can hook-into all points, not just the representative sample. Currently, we retrieve the 10k sample and show all, but, when they zoom-in to a small scale (say at the block group level) and pan, we are still looking at the representative sample, not an on-demand map service that can pipe-in the additional points.

We already have an architecture in place for the host application that performs well with big data - the maps are embedded and were added much later, but for reasons of policy, I can't make a connection to SQL Server from ArcGIS Server directly. I can, however, call Ajax methods that hit SQL Server, which is how I am getting the points. I know, I'm trying to work on changing this!

This hasn't been an issue before as our JSON point layers have been small - things have really changed with this particular data layer. If I can make a convincing argument to make a connection to SQL Server, I would like to call a stored procedure after some map action that would hit a spatial view - would this allow the same sort of on-demand caps that I can set-up for map services in the manager?

Thanks for your help.

[DanielWalton]

Maybe I don't understand all the aspects of your question, but it seems fairly straightforward that, if you have access to SQL spatial query methods in your AJAX service, you should be able to do a fresh query of the point data when the user zooms on the map using the new map extent. If the query returns more than 10k points, return the polygon data, if not, return the point data.

[ChrisSmith7]

I do not have access to SQL spatial methods - I have access to query a table with lat/long points, from which I create the JSON feature layers in JavaScript. This is difficult as I must operate within certain parameters, unfortunately. Additionally, the point layer must be made available at all times.

If this could live as a map service (e.g. exported from a feature class in ArcMap), I would be fine - I would apply a definition query, load the feature layer, and put a cap of 10k on the service through the server manager and set it as on-demand in the API.

[DanielWalton]

Can you index the Lat / Long fields? If so you can just add WHERE ([LAT] BETWEEN y1 and y2) AND ([LON] BETWEEN x1 AND x2) into a normal T-SQL query expression.

[ChrisSmith7]

Possibly. I can see this working assuming I transform the Web Mercator extent I get from the API to WGS84 and send this for filtering. One other aspect would be performance - the data layer loads in the background, and while the SQL is optimized, it's still rather slow (for web standards)... I'm not sure it would work well for a user panning/zooming. I could probably pre-aggregate a table once a day if staleness weren't a concern, so there's no complex business logic applied on-the-fly.

Thanks for the idea - I am going to investigate this some more.

[ChrisSmith7 · Accepted solution]

I ended-up creating a feature service from a permanent/flat SQL table populated by internal processes. From here, I created an xy event layer (using the lat/longs available in the table) and published this as a map service with a 1k cap. I am calling it through the JSAPI in MODE_ONDEMAND. Works very well!