How best to deal with large feature layers

[Anonymous User]

I'm currently working on getting data from a massive trail feature layer to map on an iOS app. In order to display the individual trails in different colors I thought to use the #iOS SDK filter functions to filter the data for each trails and display the filtered trail data as a different feature layer. The trail layer is massive so I see that this is taking the iOS app a sec to display the data. How do I do this filtering work remotely? Can I create new feature layers on arcgis from other layers remotely? Can this be done on a #ASP.NET Core webapp? Also generally what's the best way to deal with large data sets on mobile apps.

[Nicholas-Furness · Accepted solution]

Hi.

There appear to be a lot of questions here, only some of which are relevant to the Runtime SDK for iOS. But I'll try to provide some guidance…

Firstly, can you explain what you mean by the "iOS SDK filter functions"? Are you referring to definitionExpression?

Typically, you might draw a set of data in different colors by using an AGSUniqueValueRenderer, but that might only work if you are not displaying a huge number of trails at a time. See this sample.

Which leads to my second question. How are you displaying these trails? Will you zoom out to display hundreds of trails at a national level, or do you focus in on specific trails or areas with just a small set of trails visible at any one time?

In the latter case (i.e. you have a set of trails that you know you will be working with), you could set the AGSServiceFeatureTable's featureRequestMode to manualCache. You would then populate the local in-memory AGSServiceFeatureTable using populateFromServiceWithParameters() and set an appropriate whereClause on the AGSQueryParameters. This will make the network request once, and then display the features from memory. As you zoom and pan around, your app won't make network requests to refresh or reload the data. Other featureRequestModes might end up displaying from memory, but features are not aggressively cached so panning and zooming is likely to result in repeat requests (see the bufferFactor property for some control). This way, you have complete control over when the features are retrieved from the service.

Setting the featureRequestMode to manualCache and calling populate with a whereClause is fundamentally very similar to using definitionExpression with the default featureRequestMode, except that you have total control over how long features are cached locally and when network requests are made.

As for your other questions:

• I'm not sure what you mean by "ASP.Net Core webapp".
• You can create Feature Layer Views of a hosted feature service. I do not know how you would create and manage those remotely and it doesn't sound like it would solve your issue.
• The best way to deal with large data sets depends on a lot of variables to do with the data - these considerations will have an impact on how you might publish your data and how you might access it (e.g. the populate approach above):
  • Will you need to see it all at once when zoomed out? If so, you could publish the feature layer to a tiled image service and display the tiles when zoomed out.
  • Will you be editing it? If not, you could publish a separate generalized layer for use when zoomed out.
  • How reliable is your network connection? If it's slow or variable, you could consider the populate approach above or you might consider generating a Mobile Map Package with ArcGIS Pro and using that.

Hope some of this helps, but I'd need to understand more about what you're trying to achieve.

[Anonymous User]

Thank you so much for taking the time to reply this question, the information your provided is extremely helpful!! Apologies for how confusing the question is, my team mate and I are still learning how to best use the Esri tools. Here is our goal:

• Display the heritage trails in Oakville Ontario Canada in different colors on an a web app and on a iOS app.
• Do this in the most efficient way possible on the iOS app

The data we are leveraging is this feature service, the data doubles as a record of all sidewalks or pedestrian walkways in the town and a record of all trails, so it has many features. Each feature (sidewalk) has a field that specifics the trail it is part of. Our approach right now is

• Filter out the features from the feature service that are part of that trails we want using the field
• Display this filtered portion of the data as the different trails on a map (I guess this is where the  AGSUniqueValueRenderer comes in)

The data has a few internal use-cases within the town's various departments and was not intended to be used exactly as we intend to use it. We felt then that the safest route is to filter exactly what is needed to be displayed as best as possible. We hope that this approach we also prevent the iOS app from getting excess data. What do you think? 

[Nicholas-Furness]

Hi there.

Thanks for sharing the feature service. It turns out that this isn't very much data at all (I had imagined you had thousands of trails across a whole country/continent). Looks like the biggest trail only has 50ish features. This is not a lot at all. See the graph here.

Perhaps the easiest way to do this, since you are also going to build a web component around it, is to create a layer in ArcGIS Online that uses that service, style it, and reference that in both your web map and the iOS app. Something like this:

1. From the Open Data URL you provide above, get the API URL: https://maps.oakville.ca/oakgis//rest/services/ArcGISOnline/AGO_Oakville_Misc_Layers/MapServer/0
2. Sign in to ArcGIS Online and open a new map: https://www.arcgis.com/home/webmap/viewer.html?useExisting=0 
3. Add Data (Add Layer from Web) and provide the API URL.
4. Change the style of the new layer…
   
   …to style by Trail Network.
   
   You can explore the OPTIONS panel to change the symbols some more.
5.  Save the styled layer
   
6. This will create a new layer item in your ArcGIS Online account which references the Oakville GIS server. Go to the new layer's details page and get its ID from the URL…
   
   I also shared the new layer item publicly at this point so my iOS app wouldn't have to deal with authentication. This may or may not be what you want to do.
7. Use the Item ID from step 6 to open the layer in your iOS app. The Runtime will read the styles you defined in the map. If you later update the layer and save it, your iOS app reflect that update:
   

   import UIKit
   import ArcGIS
   
   class ViewController: UIViewController {
   
       @IBOutlet weak var mapView: AGSMapView!
       
       let featureLayer: AGSFeatureLayer = {
           let savedLayerItem = AGSPortalItem(portal: AGSPortal.arcGISOnline(withLoginRequired: false),
                                              itemID: "6d3278ab3dcf4fa28bef288fd19ddc04")
           let layer = AGSFeatureLayer(item: savedLayerItem, layerID: 0)
           
           layer.definitionExpression = "TRAILNETWORK <> 'NULL'"
           
           return layer
       }()
       
       override func viewDidLoad() {
           super.viewDidLoad()
           // Do any additional setup after loading the view.
           
           let map = AGSMap(basemapType: .lightGrayCanvasVector,
                            latitude: 43.46090487171449,
                            longitude: -79.71849182158695,
                            levelOfDetail: 12)
           map.operationalLayers.add(featureLayer)
           
           mapView.map = map
       }
   }‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

   Note that in the code above, I use the definitionExpression I mentioned in my previous email to filter the features returned for the feature layer. There is not a lot of data in this layer once you filter out the non-trail records.

You can use this same layer in your web app and in the iOS app if you need, or create one for each. In this workflow, you're still leaning on the source data API. If Oakville's servers get slammed, you might consider downloading the Shapefile of the data and uploading that to ArcGIS Online to host your own copy of the data in our cloud. These doc links might help:

• Style categories (unique values, step 4 above)
• Share items (I ignored security in the code above by sharing my layer publicly)
• Copy and save layers (step 5)
• If you need to host it in ArcGIS Online: Publish hosted feature layers (and manage them)

Hope this helps. With this data you shouldn't need to do anything special from the iOS side of things. My only concern would be whether the Oakville GIS server has good enough uptime. Probably, but if not, just upload to ArcGIS Online and keep your service updated as they publish new data.

Hope this helps.

Nick.

[Anonymous User]

This very helpful. Thank you very much.