Using definitionExpression, Filter, Effect, or * to change features on map - Large Data

[AndrewMurdoch1]

Good Day

I have a layer with ~790k features which is comprised of 15 800 features over 50 years, 15 800 * 50 = 790k.    At any one point I only need to show 1 year or 15 800 features, and the field I need to filter on is called "Year".  I'm using a Hosted Feature Layer and my current process is:

1. Grab the Feature Layer by Portal ID
2. Run a query on that layer give me all features
3. Use those features to build the render settings
4. Set the initial definitionExpression to the 2022, so I see the first batch of 15 800 features
5. Add that layer to the map:

this._layers = [];
this._layers.push(new FeatureLayer({
  portalItem: {
    id: "<feature id>"
  },
}))

this.buildRenderSetting().then((render) => {

  if (_.isEmpty(this._year)) {
    this._year = new Date().getFullYear();
  }

  this._layers[0].renderer = render;

  if (this._year) {
    this._layers[0].definitionExpression = 'Year = ' + this._year;
  }

  this._map.removeAll();
  this._layers.forEach((layer) => {
    this._map.add(layer);
  })
})

This works great, and I get a nice map with the first 15 800 features, the problem is when I change the year! 

My code to handle the year change:

if (changes.year.currentValue) {
    console.log('Year Changed');
    console.log(changes.year.currentValue);
    this._view.when(() => {
        this._view.map.layers.forEach((layer) => {
            layer.definitionExpression = 'Year = ' + this._year;
        })
    })
})

 
When that code executes I'm getting a really latent update, that results in several tile errors, and the map locks up for a solid minute or two, before eventually updating.  If I'm zoomed in to a small area ~200 features, the update is much faster, on the order of seconds.

I've tried to use both FeatureFilter and FeatureEffect, but those run so slow that it's honestly unusable,  and I can replicate the same experience with Effect / Filter on ArcGIS Online with this same layer.

I'm using a UniqueRenderValue for the render setting, looks like:

const uniqueRenderSettings = {
	type: 'unique-value',
	valueExpression: `When(${_valueExpression})`,
	uniqueValueInfos: [{
		value: <renderKey>,
		label: <renderKey>,
		symbol: {
                	type: <symbol>,
                	color: <colour>,
                	size: 12,
                	width: 3,
                	outline: {
                  		width: 4,
                  		color: <symbol>
  			}
  		}
	}]
}

Is there a way to properly handle this case?

Thanks

[JayakumarPD]

If data is large, may be an issue.  Pls check this it may help 

https://developers.arcgis.com/javascript/latest/sample-code/featurelayer-query-pagination/

[AndrewMurdoch1]

Thanks!  I'm going to try and build a version of this to load the data in parts, that might solve my issue.

I'll let you know if it works 🙂

[AndrewMurdoch1]

Thanks for the suggestion but it's still a choppy mess, no matter how I breakdown the queries.  Using pagination the year change code is now:

  if (changes.year.currentValue) {
    const useDefinitionExpression = false;
    if (useDefinitionExpression) {
      this._view.map.layers.forEach((layer) => {
        if (layer.type === 'feature') {
          layer.definitionExpression = 'Year = ' + this._year;
        }
      })
    } else {
      const featureBreak = 10;

      this._view.when(() => {
        this._view.map.layers.forEach((layer) => {
          if (layer.type === 'feature') {
            this.queryFeatureCount(layer).then((featureCount: number) => {
              const runQueryCode = true;
              if (runQueryCode) {
                const featureGroup =  featureCount / featureBreak;
                const featureSteps = [];
                for (let i = 0; i <= featureCount; i += featureGroup) {
                  featureSteps.push({
                    start: Number(i.toFixed(0)),
                    end: Number((i + featureGroup - 1).toFixed(0))
                  })
                }

                const runFeatureStepLoop = true;
                if (runFeatureStepLoop) {
                  this._map.removeAll();
                  featureSteps.forEach((step) => {
                    const query = {
                      start: step.start,
                      num: step.end,
                      where: 'Year = ' + this._year,
                      outFields: ["*"],
                      returnGeometry: true,
                    };

                    console.log('Query');
                    console.log(query);

                    layer.queryFeatures(query).then((queryResult) => {
                      try {
                        console.log('Adding New Features');
                        this._map.addMany(queryResult?.features);
                      } catch (error) {
                        console.log(error);
                      }
                    }).catch((queryError) => {
                      console.log('Error Querying Features');
                      console.log(queryError);
                    })
                  }) // END_FOREACH: Feature Steps
                }
              }
            });
          }
        })
      })
    }
  }

 
Using "featureBreak" I can control how many queries are generated, but no matter what I set that to I run into the same problem.  Right now that problem is a lot of shuddering and crashing.

Apart from using this method, are there any considered best practices?  I'm thinking we might have to use 1 layer / year to make this effective, because nothing I try seems to work.

Thanks 😀

[mgeorge · Accepted solution]

@AndrewMurdoch1 if you only have 15,800, with different time values, it might be worthy trying out the "fat table" approach, where you include the values that vary with time as different attributes in your 15,800 features. You can then use an arcade expression + visualVariable to show which time data you would like:

Here's an example of this: 

Update a renderer's attribute | Sample Code | ArcGIS API for JavaScript 4.24 | ArcGIS Developers

And the associated service metadata: Layer: GlobalTemp_AnnualMean (ID:0) (arcgis.com)

I think this One Ocean (ekenes.github.io) is using a similar approach (but depth instead of time). 

[mgeorge]

Also might be worth checking out Best Practices for Building Web Apps that Visualize Large Datasets [2021] - YouTube

[AndrewMurdoch1]

I changed the data so we only use 1 set of ~15 800 features and use the attributes to show the time values.  Using this method I have ~70 attributes, but I can use "valueExpressions" to change the render setting, giving me almost instance response and feed back because I don't have to regrab the data, I only have to apply a new render setting.

I've been extended this idea out, over the last several days, so I have a Unique Value Renderer with ~170+ settings in it, and then change the fields / "valueExpression" to change the view, which has been game changer.

If anyone else is running into this problem, try to flatten the data out and use attributes instead of duplicating features if you can get away with it 🙂

Thanks for the suggestion, I was actually logging to update my progress, but you said exactly what I did, and it worked like a charm.

Cheers

[mgeorge]

Great! Glad to hear it's working out for you.