topic Re: Who can make this ESRI'S reply simple and easy to understand ? in ArcGIS Online Questions

Who can make this ESRI'S reply simple and easy to understand ?

admedahmeds — Thu, 12 Jul 2012 18:05:24 GMT

Hello,

First of all, I would like to thank ESRI team for their great support on this forum.

I believe, here are a lot of people are asking : how can we import and upload a layer to Arcgis online with more than 1,000 feature? it seems there is only one solution for this problem, and this solution has been mentioned on this thread by ESRI Team, BUT this solution is not clear for me! so who can help me please to get the way which will help me to upload and import a layer with more than 1,000 feature?

Thanks.

http://forums.arcgis.com/threads/41778-1-000-feature-limit-in-ArcGIS-online

Re: Who can make this ESRI'S reply simple and easy to understand ?

R_C_Barenchik — Thu, 12 Jul 2012 18:30:14 GMT

If you are using 10.1, you can create an organization on ArcGIS.com. You can then create your own map on your desktop and create a map service and upload and have it hosted on arcgis.com.

Re: Who can make this ESRI'S reply simple and easy to understand ?

MikeMinami — Thu, 12 Jul 2012 23:10:14 GMT

The simple answer is you can't upload more than 1000 features into a web map (e.g., via a shapefile). Instead, you need to publish a web service of your data. From the ArcGIS world, you can do that by either publishing to your own ArcGIS Server or publishing to ArcGIS Online via a subscription.

There are two types of hosted web services that ArcGIS Online currently can create. One is a tiled map service and the other is a feature service. A tiled map service is a representation of your data in the form of snapshot images taken at set scales. The advantage of a tiled map service is that it's very fast. The data has already been drawn and stored as images on the server. When someone views your data, the server simply returns the set of images that correspond to the area being viewed. A feature service, on the other hand, will return the set of features (coordinates and attributes) for the area being viewed. Why would you use one or the other or both depends on what you need to do. Here's a quick summary of capabilities.

ARcGIS Onlne hosted tiled Map services
Are Fast
Support advanced cartography because the drawing is done by the server
Don't support queries without an associated feature service

ArcGIS Online hosted feature services
Support editing
Support coordinate and attribute queries
Are drawn by the client (thus support dynamic mapping)

Hope this helps,

Mike

Re: Who can make this ESRI'S reply simple and easy to understand ?

admedahmeds — Fri, 13 Jul 2012 10:59:39 GMT

rcbarenchik1 & Mike,

Thank you very much for your replies, the information was very helpful.

But I couldn't do it, because I think I don't have the (right) version of the Arcgis!

I couldn't find the: File > Share As > Service.

Now, my question is there a difference between ArcGIS 10, which I have, and ArcgGIS 10.1 ??

and if yes, how can I get a trial version of ArcGIS 10.1 ?

Finally, can I divide my shapefile which has around 50,000 feature to 50 shapfiles and import them to ArcgGIS online ?

Thank you once again.

Re: Who can make this ESRI'S reply simple and easy to understand ?

MikeMinami — Fri, 13 Jul 2012 21:41:57 GMT

You need ArcGIS 10.1 to publish a service in ArcGIS Online.

Can you divide your shapefile of 50000 features into 50 shapefiles? No you can't. Since you asked, here's why...

There are two reasons why you can't add more than 1000 features to a web map. One is a performance issue and the other a technical limitation of client applications, like web browsers and phones.

When adding one or more shapefiles to the web map, all the features in the shapefiles get stored in the web map. Adding a lot of data to a web map can make it large. The practical limits to web map size are 5-10 MB, but this also depends on capabilities of the client application viewing the web map as well. Web maps get downloaded to the client, so a large one can take a while to download, especially on mobile devices. So you don't really want your web maps to get too big because people will then think your map is slow.

The shapefile features stored in the web map actually get drawn as graphic elements by the client. There are limits to a client's ability to draw these graphics. In the web browser (most notably older verisons of Internet Explorer) ArcGIS.com displays an error message when the map doesn't draw completely because you've reached the limit of the browser's capability to draw graphics. This is why we recommend generalizing your data before adding it to a web map. Every vertex of every polygon gets stored and thus has to be drawn by the client. There can be a lot of unnecessary vertices depending on the scale you're viewing your data. Excess vertices just cause the client to reach its drawing limits sooner. So in reality, the 1000 feature limit is an general limit, not a hard limit enforced by ArcGIS.com.

Sometimes people mistakenly think the limit is 1000 features per shapfile and proceed to add 3 shapefiles, each with 1000 features. This is incorrect because it's actually the client's ability to draw the features, independent of how many shapefile layers there are. The tricky part is that different clients have different limits. A web map may display fine in Google Chrome and Firefox, but not in older versions of Internet Explorer.

The solution, as I wrote in a previous post is to create a web service of your data--either a map or feature service (optionally a hosted service on ArcGIS Online if you don't have ArcGIS Server). However, I will point out that the same client drawing limit applies to feature services (both hosted and ArcGIS Server), and this is where some confusion may arise. While a feature service itself is not limited in the number of features it can store, the features are still drawn by the client and subject to the same drawing limitation as when the features are stored in the web map. That's why you need to be very thoughtful about setting scale dependencies on your feature services. For example, you don't want to view a parcel database when zoomed out at the world level. It's just too many features to request from the server (there are limits on the size of queries the server will accept too) and too many features for the client to draw.

Thus, a feature service may not the best solution, especially for displaying large amounts of data on the client. If you need this ability, you should probably be using a map service, and for maximum performance a tiled map service. However if you need to support editing, then you have to use feature services. But, you need to configure the properly to perform well.

Thanks,

Mike

Re: Who can make this ESRI'S reply simple and easy to understand ?

SauloGustavo_de_Oliveira — Thu, 25 Feb 2016 01:28:23 GMT

Hello Mike,

This post is old but is excellent! Congrats! I would like to ask a couple of directions.

Does the limits of features still at 1000 features in ArcGIS Online, right? Where can I found the documentation or articles about this?

What´s is the better aproach when we need to work with a data with more than million features.

Thank you,

Re: Who can make this ESRI'S reply simple and easy to understand ?

MikeMinami — Fri, 04 Mar 2016 00:11:20 GMT

The 1000 features is not a hard limit, but an easy way to explain this. Obviously, drawing 1000 points as graphic elements is much less work than drawing 1000 polygons, each with 1000 vertices.

As far as the best approach, that's a hard question to answer. Also there's a difference between storing your data on premises (Portal for ArcGIS) or in the cloud (ArcGIS Online).

ArcGIS Online provides two ways to serve our your vector data:

- Hosted feature layer

- Hosted tile layer

Portal for ArcGIS (which includes ArcGIS for Server) has another way

- Feature layer

- Tile layer

- Map image layer (aka Map Service)

In the previous post, I describe the advantages between hosted feature and hosted tile layers. If you're using Portal, you have the Map Image Layer option, which is sort of a blended case. With a map image layer, the server does all the drawing and returns an image of the layer to a client. But if you need to, you can access the actual geometry of a feature.

Hope this helps,

Mike

Re: Who can make this ESRI'S reply simple and easy to understand ?

CharlesBuzzard — Thu, 29 Nov 2018 16:20:26 GMT

Since this discussion began in 2012 and the answer from Esri really hasn't changed over the life of this topic, I would suggest Esri re-examine this issue. In 2012 most mobile devices came with 16 GB of storage and 1 GB of RAM while today 64 GB and 4 GB are the minimums Comparison of smartphones - Wikipedia In 2012 only large cities had access to 4 G cell networks while today over 50% of world population has access to 4 G networks Mobile broadband - Wikipedia . 5 G is now available in some large cities and should be widespread by 2020.

Limiting the number of features passed a client device made sense in 2012, but to not keep up with the changes in network speeds and devices, doesn't fit Esri's marketing image. We understand there are other ways to pass maps and data to client devices than to load shape files, geo JSON files or CSVs, but since Esri makes these functions available we need to Esri to keep this functionality current with technological changes.

We are looking for Esri to keep up with industry changes by increasing these limitations and if they want to make suggestions to developers for certain devices we would welcome their input. Developers should be able to limit feature throughput based on our client audience not an antiquated technology spec from 5 or 6 years ago.

While this is a little bit of a rant, this is meant to be an encouragement to keep a technology many of us depend on current. While Esri's developer resources have been focused on other products, it's time to redirect some efforts in this area.

Chuck Buzzard, MS, GISP

GIS Supervisor, Spatial Services Section

Finance Department, Technology Division

1102 Broadway, Suite 101

Tacoma, WA 98402

253-798-7703

cbuzzar@co.pierce.wa.us

Re: Who can make this ESRI'S reply simple and easy to understand ?

MikeMinami — Thu, 29 Nov 2018 18:15:13 GMT

Hi Chuck,

Esri has continually advanced our performance and drawing of hosted layers since the original post in 2012. For example, we've continually evolved how we request large amounts of features from large hosted feature layers, we leverage client side, server side and CDN (Content Delivery Network) caching, and can use WebGL (Web Graphics LIbrary) when drawing layers. Here are a few blog posts I found that describe these enhancements.

https://community.esri.com/community/gis/web-gis/arcgisonline/blog/2017/10/17/so-you-want-to-display-a-bazillion-features-on-the-web-a-technical-approach-to-large-feature-datasets

Scalable hosted feature layers in ArcGIS Online: Tile queries and response caching

FeatureLayer rendering: taking advantage of WebGL in 2D

From your post, it seems like you may be loading shapefiles and CSV files directly into web maps. While we have made some performance improvements over the years to increasing the size of shapefiles stored in web maps, we will never be able to get the same level of performance as we do with hosted feature layers. It's just not technically possible.

Is there something preventing you from publishing your data as hosted layers? This is really the recommended way to go.

Mike

Re: Who can make this ESRI'S reply simple and easy to understand ?

CharlesBuzzard — Thu, 29 Nov 2018 22:01:03 GMT

Mike,

Thanks for getting back to me on this.

We have users that do not have ArcGIS Online accounts and our agency will not allow them to put their ad hoc data on our servers. They create shape files by extracting our data or from other software packages and need to load these features into our applications to display and query their data. They could use QGIS or some other GIS app, but since our agency has the most complete data respository, they are interested in using our apps for this purpose. We had an old ArcIMS application that allowed them to do this, but the 1000/4000 feature limit of the ArcGIS environment is a set back to say the least. We have been able to improve most functions by moving to the new Esri solutions, but this limitation is not an improvement in our client's eyes.

I hope you can pass this along as an enhancement for a much needed future release.

Thanks,

Chuck

Re: Who can make this ESRI'S reply simple and easy to understand ?

MikeMinami — Fri, 30 Nov 2018 21:29:00 GMT

Chuck,

I'll pass along your comments.

About a year ago, we did increase the limits for uploading shapefiles. This is from the documentation.

Shapefiles—ArcGIS Online Help | ArcGIS

The shapefile must be less than 10 MB in size and have less than 4,000 point features or 2,000 line or polygon features. If the file is too large, generalizing the features will often reduce the overall size and allow the shapefile to be added to the map. If that doesn't work and you have publishing privileges in your organization, it is recommended that you publish shapefiles that exceed these limits as a hosted feature layer.

My understanding is that we've kind of reached the limit on the amount of data we can store inside a web map. Storing all of that data inside the map makes the map quite large. That 10 MB shapefile is stored as plain text (JSON) inside the web map and probably yields an even larger map than the original shapefile. That map has to be downloaded to run on the device. Maybe it could perform OK on a PC on a fast internal network, but, for example, downloading that to a mobile device would likely be slow.

You say that people download parts of your data. Is your data hosted in AGOL? Are there some common areas or parts of your data that many clients might be interested in using? One idea is to create view layers from a hosted feature layer. View layers can define a subset of features and attributes from the original layer. So, given your source data, you might make several view layers that are subsets that people frequently use.

Create hosted feature layer views—ArcGIS Online Help | ArcGIS

Set hosted feature layer view definition—ArcGIS Online Help | ArcGIS

Hope this helps,

Mike

Re: Who can make this ESRI'S reply simple and easy to understand ?

CharlesBuzzard — Fri, 30 Nov 2018 21:47:01 GMT

Mike,

Thanks for passing this along. Several of us have read through the documentation you referenced and we have tried the recommendations, but having clients who want to load ad hoc data keeps us from implementing the recommendations. Being able to control the data inputs is how we operate in most cases, but we really need to support this use case. Not best practice, but a need in our world.

Thanks again

Chuck