Dashboard Expression Performance Improvement Needed

[Caitlin_Todd_LCOR]

Hi all, 

I'm hoping this would be an easy Yes or No answer. Is there a way to rewrite this expression so performance speed could be improved? Currently it does run correctly, but takes 7-10min to finish. End result will be a count of failing inspections within high priority features in an Indicator widget in ArcGIS Dashboard.

Thanks!

Caitlin

// set portal variable and feature set of failing ramp inspections
var portals = portal('Some org url');
var mRamp = FeatureSetByPortalItem(portals, 'some item id',8,['PassFail','CalibrationDate','latitude','longitude','GlobalID'], true);
var sql =  `PassFail = 'Fail'`;
Var FailRamps = Filter(mRamp,sql);

//set feature sets of prox priorities
var Parks = FeatureSetByPortalItem(portals, 'some item id',4,['name','approx_acres','GlobalID','latitude','longitude'],true);
var GovFacility = FeatureSetByPortalItem(portals, 'some item id',8,['GlobalID','MAPTAXLOT','STATCLDES','AscendAcres'],true);
var PublicFacility = FeatureSetByPortalItem(portals, 'some item id',7,['GlobalID','MAPTAXLOT','STATCLDES','AscendAcres'],true);
var CareFacility = FeatureSetByPortalItem(portals, 'some item id',9,['GlobalID','MAPTAXLOT','STATCLDES','AscendAcres'],true);
var SchoolFacility = FeatureSetByPortalItem(portals, 'some item id',6,['GlobalID','MAPTAXLOT','STATCLDES','AscendAcres'],true);
var ZoningLayer = FeatureSetByPortalItem(portals, 'some item id',20,['ZONE_NAME','GlobalID'],true);

//add FailRamp feature to captured ramps counter when it is within 0.125 miles of a priority facility
var capturedRamps = 0;
for(var f in GovFacility){
  var govbuff = BufferGeodetic(f, 0.125, 'miles');
  If(Count(Intersects(FailRamps,govbuff))>0) {
    capturedRamps ++;
  }
}

for(var f in CareFacility){
  var carebuff = BufferGeodetic(f, 0.125, 'miles');
  If(Count(Intersects(FailRamps,carebuff))>0) {
    capturedRamps ++;
  }
}

for(var f in PublicFacility){
  var pubbuff = BufferGeodetic(f, 0.125, 'miles');
  If(Count(Intersects(FailRamps,pubbuff))>0) {
    capturedRamps++;
  }
}

for(var f in SchoolFacility){
  var schlbuff = BufferGeodetic(f, 0.125, 'miles');
  If(Count(Intersects(FailRamps,schlbuff))>0) {
    capturedRamps ++;
  }
}

for(var f in Parks){
  var parkbuff = BufferGeodetic(f, 0.125, 'miles');
  If(Count(Intersects(FailRamps,parkbuff))>0) {
    capturedRamps ++;
  }
}

for(var f in Zoninglayer){
  If(Count(Intersects(FailRamps,f))>0) {
    capturedRamps ++;
  }
}

return capturedRamps

[jcarlson]

It looks like you've got your FailRamps featureset, and you're iterating through every feature in 6 layers and checking for spatial intersection, then adding to the count. If a failed inspection is within 0.125 miles of multiple locations across the 6 layers, won't you be counting inspections multiple times?

I would instead try to iterate through the FailRamps featureset and check for spatial intersections with the 6 layers, and as soon as one is found, increment the count by 1 and move to the next inspection. That way the resulting count will be the number of inspections that are within 0.125 miles of any of those 6 layers' locations, which sounds more like what you want.

Supposing all 7 layers involved had 100 features. The way you're doing it is to run 600 buffers and 600 intersections. The proposed alternative would be to run 100 buffers and at most 600 intersections, though probably less. It will still be a costly expression, but it might evaluate a bit faster.

Granted, I don't know what the feature counts in your layers look like, but I think it's worth a shot.

Roughly, that might look like

var capturedRamps = 0
for (var f in FailRamps) {
    var rampbuff = BufferGeodetic(f, 0.125, 'miles')
    if (Count(Intersects(rampbuff, GovFacility)) > 0) {
        capturedRamps ++
        break
    } else if (Count(Intersects(rampbuff, CareFacility)) > 0) {
        capturedRamps ++
        break
    }
    ... and so on
}

return capturedRamps

[JohannesLindner]

• You don't need all those fields. Loading many fields slows the expression down. You actually don't need any fields for your calculation, so you should be a able to get away with only loading GlobalID.
• Reverse your intersect.
  • Right now, you're iterating over each feature of each priority fs, buffer it and check for intersections with the ramp fs.
  • Instead, iterate over the ramp fs, create a buffer for each ramp and check for intersections with each priority fs. This means you call Buffer() and Intersects() far less.
  • If you do it that way, you also don't need to get the priority fs geometries (which also slows the expression down) because you don't actually need to work with the geometries. The Intersects() is done by AGOL/Portal

var portals = portal('Some org url');
var mRamp = FeatureSetByPortalItem(portals, 'some item id',8,['PassFail', 'GlobalID'], true);
var sql =  `PassFail = 'Fail'`;
Var FailRamps = Filter(mRamp,sql);

//set feature sets of prox priorities
var PriorityFeaturesets = [
    FeatureSetByPortalItem(portals, 'some item id',4,['GlobalID'], false),
    FeatureSetByPortalItem(portals, 'some item id',8,['GlobalID'], false),
    FeatureSetByPortalItem(portals, 'some item id',7,['GlobalID'], false),
    FeatureSetByPortalItem(portals, 'some item id',9,['GlobalID'], false),
    FeatureSetByPortalItem(portals, 'some item id',8,['GlobalID'], false),
    FeatureSetByPortalItem(portals, 'some item id',20,['GlobalID'], false),
]

var capturedRamps = 0
for(var fr in FailRamps) {
    var frBuffer = Buffer(fr, 0.125, "miles")
    for(var i in PriorityFeaturesets) {
        capturedRamps += Count(Intersects(PriorityFeaturesets[i]))
    }
}

return capturedRamps

It could very well be that this doesn't make the expression much faster. It turns out that accessing a featureset for the first time just takes a lot of time, while subsequent accesses are basically instantaneous. Consider lending your support to this idea to maybe get that changed in the future.

[Caitlin_Todd_LCOR]

Hi Josh and Johannes, 

Thank you both for helping me figure this out! It's proving to be more complicated than I originally thought. After reading up on the Indicator widget in ArcGIS Dashboard I saw that it needs to have a featureset returned not a count of features. So starting out with what you created I'm trying to alter it to return a featureset instead of a capturedramps count. But I'm still unsuccessful. 

The end result will hopefully give me a featureset of all failing ramp inspections that are within 0.125 miles of priority features OR within the county zoning layer. 

This is what I have now, skipping past the portal set up and priorityfeatures variables.

var capturedramps = []
for(var f in FailRamps) {
  var fbuff = Buffer(f,0.125,'miles')
  for (var i in PriorityFeatures) {
    IF(Intersects(fbuff,i)==true) {
      Push(capturedramps,f)
    }
    else if (Intersects(ZoningLaneCounty_Prox,f)==true) {
      Push(capturedramps,f)
    }
  }
}
return FeatureSet(capturedramps)

This might be the right direction to go in? Or it might not be.. 

Thank you and have a good day!, 

Caitlin 

[JohannesLindner · Accepted solution]

Right, this was another question where Josh's very helpful blog post can help (for performance, not for your new problem).

To return a Featureset, you first have to define it (what fields and geometry type does it have?), and then you have to fill it.

Something like this should do the trick (Note the use of Memorize() for all loaded Featuresets, this should hopefully help with performance a lot.):

// function to load a Featureset into RAM
function Memorize(fs) {
    var temp_dict = {
        fields: Schema(fs)['fields'],
        geometryType: Schema(fs).geometryType,
        features: []
    }
    for (var f in fs) {
        var attrs = {}
        for (var attr in f) {
            attrs[attr] = Iif(TypeOf(f[attr]) == 'Date', Number(f[attr]), f[attr])
        }
        Push(
            temp_dict['features'],
            {attributes: attrs, geometry: Geometry(f)}
        )
    }
    return FeatureSet(Text(temp_dict))
}


// load your Featuresets
var portals = ...
var mRamp = Memorize(FeaturesetByPortalItem(...))
var sql = "PassFail = 'Fail'"
var FailRamps = Filter(mRamp, sql)

var PriorityFeaturesets = [
    Memorize(FeaturesetByPortalItem(...)),
    // ...
]

var zoningLayer = Memorize(FeaturesetByPortalItem(...))


// define the output featureset
var out_fs = {
    geometryType: "",
    fields: [
        {name: "GID", type: "esriFieldTypeString"},
    ],
    features: []
}

// fill the featureset with all ramps that are near priority features or wihtin the zoning layer
for(var ramp in FailRamps) {
    var zone = First(Intersects(zoningLayer, ramp))
    if(zone != null) {
        Push(out_fs.features, {attributes: {GID: ramp.GlobalID}})
        continue
    }
    var rampBuffer = Buffer(ramp, 0.125, "miles")
    for(var i in priorityFeaturesets) {
        var priorityFeature = First(Intersects(priorityFeatures[i], ramp_buffer))
        if(priorityFeature != null) {
            Push(out_fs.features, {attributes: {GID: ramp.GlobalID}})
            break
        }
    }
}

// return the output Featureset
return Featureset(Text(out_fs))

The way I wrote it, it should not count ramps that are near to multiple priority features multiple times. To be sure, you could just throw a Distinct() around the returned fs:

return Distinct(Featureset(Text(out_fs)), ["GID"])

[Caitlin_Todd_LCOR]

Hi Johannes, 

That works perfectly! Thank you so much. Also thank you for the link to Josh's blog post. That's the first I've seen of 'Memorize' and for storing things in RAM for Arcade expressions. I hope to hear more about it in the future as it's incredibly helpful in speeding up performance! Instead of taking several minutes churning through everything, it finished in just a minute!

I'll certainly be using it more in the rest of my dashboard projects as well.

Caitlin

[JohannesLindner]

That sounds great! Please also consider kudoing this idea to hopefully have functionality like that in the native Arcade functions one day.

[DougBrowning]

I was surprised by this also.  I actaully talked to Paul at Esri and sent him my code showing how it keeps making calls to the service inside loops.  He confirmed that a var pointing to a FeatureSet does not load anything into memory like most lauguages do.  It is just a pointer so that it waits until you need it.  We then disagreed if this is good or bad.  But sounds like it will not change.  

Where I really saw it is lopping on a featureset and applying a filter in the loop.  Totally grinds to a halt.

So instead what we are doing is loading what we need into an array.  The array is actually in memory and its way faster.  

Here is an example.  We are looking to see which records in one table also have records in a second table (a location and any known errors or not).  Since only the Key is needed to check for existence you can push the key into an array then check against the array vs the featureset.   sorry the formatting is from an email so its a mess

code that you would think would be right

//Cycles through each record in the input table which should be in memory
for (var f in tbl) {
    
    //Determine if there are any unresolved errors
    var sql = "PointID = '"+ f.PointID + "' And ResponseType = 'Log an issue' And (Resolved IS NULL Or Resolved = 'No')"; 

// filter tbl2 based on the ID for tbl.  It seems to be making a call every time here.  This code is super slow and usually times out.
    var tbl2 = Filter(tbl2All,sql);
    if (count(tbl2) > 0) {var v_CountUnresolved = "Yes"}
    else {var v_CountUnresolved = "No"}; 

What works better

Pushing the values into an Array then looping on that is much faster.  Open to more ideas here. 

var p = 'https://arcgis.com/'; 

var tbl = FeatureSetByPortalItem(Portal(p),'713e3aaef7d333b618',0,['Project','PointID','StreamName','PointType','OrderCode','EvalStatus','Trip'],true); 

 

//This is the schema I want to append data into. 

var Dict = {   

    'fields': [{ 'name': 'Project', 'type': 'esriFieldTypeString' }, 

            { 'name': 'PointID', 'type': 'esriFieldTypeString' }, 

            { 'name': 'StreamName', 'type': 'esriFieldTypeString' }, 

            { 'name': 'PointType', 'type': 'esriFieldTypeString' }, 

           { 'name': 'OrderCode', 'type': 'esriFieldTypeString' }, 

            { 'name': 'EvalStatus', 'type': 'esriFieldTypeString' }, 

            { 'name': 'Trip', 'type': 'esriFieldTypeString' }, 

            { 'name': 'CountUnresolved', 'type': 'esriFieldTypeString' }],   

    'geometryType': 'esriGeometryPoint',    

    'features': []};   

var index = 0; 

 

// we only need to check for existence  

var sql2 = "ResponseType = 'Log an issue' And (Resolved IS NULL Or Resolved = 'No')" 

var tbl2All = Filter(FeatureSetByPortalItem(Portal(p),'713e3aaef9674e3493a64347d333b618',10,['PointID','ResponseType','Resolved'],false), sql2); 

var tbl2text = [] 

for (var i in tbl2All) { 

    Push(tbl2text,i.PointID) 

} 

 

var isUnresolved = '' 

//Cycles through each record in the input table 

for (var f in tbl) { 

    if (Includes(tbl2text,f.PointID)) { 

        isUnresolved = 'Yes' 

    } 

    else { 

            isUnresolved = 'No' 

    } 

     //This section writes values from tbl into output table and then fills the variable fields 

    Dict.features[index] = { 

        'attributes': {    

            'Project': f.Project, 

            'PointID': f.PointID, 

            'StreamName': f.StreamName, 

            'PointType': f.PointType, 

            'OrderCode': f.OrderCode, 

            'EvalStatus': f.EvalStatus, 

            'Trip': f.Trip, 

            'CountUnresolved': isUnresolved 

        }, 

            'geometry': Geometry(f)};    

    ++index; 

     

} 

return FeatureSet(Text(Dict)); 

 hope that helps