The RasterStretchRenderer works well when you have a large range of values to display, such as in imagery, aerial photographs, or elevation models.
ESRI Canada Technical support got back to me with this regarding performance differences between ClassBreaks and the StretchRenderer:
..the primary difference between the two is that the Class Breaks Renderer consumes more resources because it must query the attribute information on the layer first before determining how best to categorize each pixel in each class. So there is a lot of computation and logic involved behind the scenes when this renderer does its work to put each pixel value into each group. This is passed to the client, and explains the performance decrease when you dynamically change this renderer.
The documentation notes this: 'ClassBreaksRenderer defines the symbol of each feature in a Layer based on the value of a numeric attribute. Symbols are assigned based on classes or ranges of data. Each feature is assigned a symbol based on the class break in which the value of the attribute falls.'
The Stretch Renderer is working less hard as it does not have to consider all attribute values and how they may be placed into groups of categories. It is simply just changing the values on pixels. This has less impact on the client, as is seen in your sample.
The documentation on this renderer notes: 'RasterStretchRenderer defines the symbology with a gradual ramp of colors for each pixel in a ImageryLayer based on the pixel value.'
Makes sense. There was also confirmation that there isn't much in the way of other options for a client-side renderer on an ImageryTileLayer. So sounds like for now we'll opt for this approach despite it's limitations and hope that the RasterStretchRenderer will be extended to filter values at some later point by ESRI.
If anyone comes up with better solutions or workarounds would love to hear.
Update: this no longer seems to work at ArcGIS JavaScript API 4.17. Reached out to ESRI support for advice.
ESRI Canada notified me that a bug (BUG-000137572) has been logged by Redlands that causes this not to work at versions 4.17+ .
A simplified CodePen showing how filtering a ClassBreaksRenderer for a raster at 4.18 fails now works using "next" (4.19).