Mapping over 2,500 points - Best practices

876
7
07-18-2017 06:01 AM
EdwardSmathers
New Contributor III

Hello - I have asked this question before with no replies, but would greatly appreciate any advice.

I have a CSV of over 2,500 points, with a couple dozen columns of information pertaining to each point. I am aware that I can add over 1,000 points to any map, but will lose performance in certain situations and uses.

Can anyone offer advice as to best practices for managing this type of content? I am able to break the data into regions within the state, but don't want to break it up if it is not necessary.

My goal is to offer an online map of the data and a mobile app.

Thanks in advance!!

Tags (1)
0 Kudos
7 Replies
TedKowal
Occasional Contributor III

I don't know if this will help, but may spur some ideas.  Currently I use Bing Maps to overlay my GIS stuff on.... (not my preference but the client's).  I have my GIS data in SQL server.  What I do for displaying  large numbers of data  points for example "Street Lights" is to query the users exent and do an intersect spatially against the GIS data only returning those street lights in the users current view.  

My SQL procedure for sending back only data found in the extent sent to it as parameters  ul/br lat and long   (Upper Left/ Lower Right)

USE [GeoTest]
GO
/****** Object:  StoredProcedure [dbo].[usp_ColorLightLC]    Script Date: 7/18/2017 11:53:37 AM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author:          Ted Kowal
-- Create date: 6/21/2013
-- Description:     Lights and LC from Map View
-- =============================================
ALTER PROCEDURE [dbo].[usp_ColorLightLC] 
     -- Add the parameters for the stored procedure here
     @ulLat nvarchar(10), 
     @ulLong nvarchar(10),
     @brLat nvarchar(10),
     @brLong nvarchar(10)

AS
BEGIN
     -- SET NOCOUNT ON added to prevent extra result sets from
     -- interfering with SELECT statements.
     SET NOCOUNT ON;

-- Create a rectangle geography instance based on bounding box of the Virtual Earth map
DECLARE @SearchRectangleString VARCHAR(MAX);
SET @SearchRectangleString = 'POLYGON((' + @ulLong + ' ' + @ulLat + ',' + @ulLong + ' ' + @brLat + ',' + @brLong + ' ' + @brLat + ',' + @brLong + ' ' + @ulLat + ',' + @ulLong + ' ' + @ulLat + '))';

Declare @G geography;
Declare @SearchArea geometry;

SET @G = geography::STPolyFromText(@SearchRectangleString, 4326);
SET @SearchArea = geometry::STGeomFromWKB((Select @G.STAsBinary()),4326);

Select 
     Shape.STGeometryType() as GeometryType,
     Shape.STAsText() as WKT,
     'Light' as Title,
     'LoadCenterID: ' + cast(LoadCenterID as nvarchar) as Description,
     LoadCenterID,
     1 as FT
from 
     MDXLIGHTS
where
     Shape.STIntersects(@SearchArea) = 1 and LoadCenterID <> 998

union all

select 
     Shape.STGeometryType() as GeometryType,
     Shape.STAsText() as WKT,
     'LoadCenter' as Title,
     'LoadCenterID: ' + cast(LoadCenterID as nvarchar) as Description,
     LoadCenterID,
     0 as Feat
from LOADCENTERS
where
     Shape.STIntersects(@SearchArea) = 1

Order by LoadCenterID ASC, FT DESC 
END‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
EdwardSmathers
New Contributor III

Hello, Ted - Thank you for the in depth reply! I think your work around is ingenious, although I want to try to work within the constraints of the ArcGIS website and app environment. Basically, I want to start out with as little custom coding as possible and then work from the (Keep It Simple, so to speak).

My main concern is speed and also the "cleanliness" of the data being presented. I have waterfall locations as my point data. Using the streetlight comparison, looking at the state of New York (waterfalls) is like looking at a map of New York City (streetlights).

Is it suggested to place data on different layers in order to prevent over 1000 points per layer or am I looking at this all wrong?

0 Kudos
TedKowal
Occasional Contributor III

That would be a way for presentation. Divided up by regions on different layers, however the real power of GIS is data and serving up that data.  I picture a waterfall site having the ability to search by location, near by, parking, hiking, by address .....  your layers now have just become bigger...  

I am pretty sure the ArcGIS site can work with data and has geo-processing tools such as spatial intersections available to use.  Hopefully, someone here who does use the ArcGIS web site can more fully help you out in your context.

Doing a little searching...

performance - Can ArcGIS Online navigate and identify large GIS datasets? - Geographic Information S... 

Best practices for using layers in maps—ArcGIS Online Help | ArcGIS 

Perform analysis—ArcGIS Online Help | ArcGIS 

GünterDörffel
Occasional Contributor III

Honestly,

2500 points is near to nothing and should not pose any challenge even on the weakest hardware. Using the csv as event theme might not be ideal, no spatial indexing, no attribute indexes possible - so realy LODING the data into a FeatureClass might be better. 

Do you just "fear" performance issues or have you experienced them. If so, can you share the data for investigation?

Regards
Guenter

EdwardSmathers
New Contributor III

Good Morning Günter and thank you for the feedback. 

Right now I plan to load them by CSV and them transition them into a FeatureClass, just as you explained. 

You have done a lot to set my mind at ease as far as the lag in loading. I have tried to load them all and had mixed results (about 10% of the points never uploaded so I loaded them separately and merged the two tables.

Ultimately my goal is as Ted suggests; being able to look at local or nearby locations and find parking/trail heads easily for my users. I do have a lot of that data collected already, but it sounds like I will want that all on separate layers to make the interface easier for the users.

My goal is to load the right way the first time, since this data has been collected over a 15 year period, with lots of rework involved due to not following best practices in the past.

I hope this answers your questions and might help you offer up some other opportunities for my data. 

0 Kudos
GünterDörffel
Occasional Contributor III

I understand your wish to keep it easy for your users. As of today this is what everyone is looking for. That - to me - also includes the minimum number of layers needed ... So if your users need to have certain combination of data, have you ever tired to create a WebApp Builder App with a Query-Tool that holds prefefined queries. That way they choose from a list of "clear text sentences" like "All easy hikes in Region xy" and the complexity of selection has been hidden by you offering this query. You might end up with only a few layers and a few settings in the app that way ...

EdwardSmathers
New Contributor III

That is some great advice  g.doerffelmysynergis-esridist - I am going to take that into consideration when publishing the data. I believe, with that thought in mind, I will be able to publish as few as three layers (but the execution will ultimately show if this is true). Thank you, sir!

0 Kudos