Dashboard Data Expression Intersects

[LukeGilner1]

I'm trying to set up some data expressions in Dashboards to use with Indictor widgets.  I have a tracts layer (322 features) and a lakes layer (11,200 features) and I want to intersect these to get the total number of lakes and a sum of lake acres in the Indicator.  Here is the code, which appears to work, but takes about 20 minutes to run.  What can I do to speed this up?  Can I run an intersection before the loop to filter out most of the lakes?  There should only be about 1,020 lakes that actually intersect with the tracts.  I only know a little Arcade; most of this was done with the help of CoPilot.

var port = Portal("https://www.arcgis.com");

// Load layers with correct area field name
var tracts = FeatureSetByPortalItem(port, "b9fcbedac0384e32bc3dba6aec1a8cf6", 0, ["OBJECTID"], true);
var lakes = FeatureSetByPortalItem(port, "5564b2e702364aefba08df8c95216a1f", 0, ["OBJECTID", "Shape__Area_2"], true);

var fsDict = {
    fields: [
        { name: "OBJECTID", type: "esriFieldTypeOID" },
        { name: "LakeCount", type: "esriFieldTypeInteger" },
        { name: "Acres", type: "esriFieldTypeDouble" }
    ],
    geometryType: "",
    features: []
};

for (var t in tracts) {
    var tractGeom = Geometry(t);
    var lakeCount = 0;
    var totalAcres = 0;

    for (var lake in lakes) {
        if (Intersects(Geometry(lake), tractGeom)) {
            lakeCount += 1;
            totalAcres += lake["Shape__Area_2"] / 4046.8564224;
        }
    }

    Push(fsDict.features, {
        attributes: {
            OBJECTID: t.OBJECTID,
            LakeCount: lakeCount,
            Acres: Round(totalAcres, 1)
        }
    });
}

return FeatureSet(fsDict)

[KenBuja]

Doing lots of intersects is a resource hog, since the code has to make external calls for each feature in the loop. @jcarlson wrote a good post about how to store the features in memory (using his Memorize function) to speed up this process. This would be good if you wanted to create a table showing how many lakes are in each tract.

However, since you're looking to just get the sum off all the lakes in all the tracts, you can loop through the tracts FeatureSet to union them into a single feature, then use that feature in your Intersects function. You can get the count of that FeatureSet to see how many lakes there are and use the Area function to get the total area of all the lakes.

var port = Portal("https://www.arcgis.com");

// Load layers with correct area field name
var tracts = FeatureSetByPortalItem(port, "b9fcbedac0384e32bc3dba6aec1a8cf6", 0, ["OBJECTID"], true);
var lakes = FeatureSetByPortalItem(port, "5564b2e702364aefba08df8c95216a1f", 0, ["OBJECTID"], true);

var arrTracts = [];
for (var tract in tracts) {
  Push(arrTracts, tract);
}
var lakesFS = Intersects(lakes, Union(arrTracts));
return FeatureSet(
  {
    fields: [
      { name: "LakeCount", type: "esriFieldTypeSingle" },
      { name: "Acres", type: "esriFieldTypeDouble" }
    ],
    features: [
      {
        attributes:
          {
            LakeCount: Count(lakesFS),
            Acres: Area(lakesFS, "acres")
          }
      }
    ]
  }
);

[LukeGilner1]

This gives me an "Unknown Error".  I've been trying some other things, but can't figure it out.

[KenBuja]

I've tested the code using some of my own data and it returned the expected data. I did make a change to line 17 to use the field type "esriFieldTypeInteger" since Count returns an integer, but keeping it as it is also works.

The next step would be to put some debugging Console messages in the code at various places (like before and after line 11) and click the Run button in the editor window. This will give you a better idea where the code is crashing.

[MiaWhite34]

Looping through both layers is what’s slowing it down. Pre-filter the lakes using a spatial query (e.g., Intersects or Within at the feature set level) before looping. This limits iterations to only intersecting features and cuts processing time dramatically.

[LukeGilner1]

@KenBuja  - I'm not sure what the issue was, but I did change the field type to "esriFieldTypeInteger" for the Count.  I also added a Filter to the Lakes layer like @MiaWhite34 suggested, using an OrgKeyID which is only greater than 0 when it intersects with a tract now.  This minimizes the lakes to 997 features instead of over 11,000, and speeds up the process a lot.

However, now the acreage is not calculating correctly.  It is calculating the entire acreage of all the lakes, rather than just the intersected areas.  The calculation is an even larger number when I use the Area function with acres, so I went back to dividing the Shape__Area_2 field (square meters) by 4046.8564224.

var port = Portal("https://www.arcgis.com");

// Load layers
var tracts = FeatureSetByPortalItem(port, "b9fcbedac0384e32bc3dba6aec1a8cf6", 0, ["OBJECTID"], true);
var lakes = FeatureSetByPortalItem(port, "5564b2e702364aefba08df8c95216a1f", 0, ["OBJECTID", "OrgKeyID","Shape__Area_2"], true);
var lakesfilt = Filter(lakes,'OrgKeyID >0')

var arrTracts = [];
for (var tract in tracts) {
  Push(arrTracts, tract);
}
var lakesFS = Intersects(lakesfilt, Union(arrTracts));
return FeatureSet(
  {
    fields: [
      { name: "LakeCount", type: "esriFieldTypeInteger" },
      { name: "Acres", type: "esriFieldTypeDouble" }
    ],
    features: [
      {
        attributes:
          {
            LakeCount: Count(lakesFS),
            Acres: (Sum(lakesFS,"Shape__Area_2")/4046.8564224)
          }
      }
    ]
  }
);

[KenBuja]

I guess I'm not clear on the number you're expecting to get for the total area. Do you want to sum up the area of the portion of each lake that intersect the each tract? If so, then you'll have to loop through each tract and each lake to get the Intersection geometry and add its area to the total. However, the total count of the lakes will likely be incorrect, since a lake intersecting several tracts will be counted for each tract.

[LukeGilner1]

Yes, that's correct.  I want the total area of the portion of all lakes that intersect with the tracts.  I was hoping the Union on the tracts would solve the issue of double counting a lake, because I have noticed that to be a little off too.  The tracts layer has lots of multipart polygons, but this could be all merged into one polygon if that's possible in the code?

[KenBuja · Accepted solution]

OK, this should give you what you're trying to do, but I haven't tested it. It loops through each tract, gets the intersecting lakes for that tract, then gets the intersecting area for each lake. Then it gets the total lake count. Note that the area is rounded to two decimal places.

It will take a while to process. You could test whether "Memorizing" the tracts and lakes layers makes it any faster.

var port = Portal("https://www.arcgis.com");

// Load layers
var tracts = FeatureSetByPortalItem(port, "b9fcbedac0384e32bc3dba6aec1a8cf6", 0);
var lakes = FeatureSetByPortalItem(port, "5564b2e702364aefba08df8c95216a1f", 0, ["OrgKeyID"]);
var lakesfilt = Filter(lakes, "OrgKeyID > 0");

var totalAcres = 0;
var arrTracts = [];
for (var tract in tracts) {
  var intersectingLakes = Intersects(lakesfilt, tract);
  Push(arrTracts, tract);
  for (var intersectingLake in intersectingLakes) {
    var lakePart = Intersection(tract, intersectingLake);
    totalAcres += Area(lakePart, "acres");
  }
}

var lakeCount = Count(Intersects(lakesfilt, Union(arrTracts)))

return FeatureSet(
  {
    fields: [
      { name: "LakeCount", type: "esriFieldTypeInteger" },
      { name: "Acres", type: "esriFieldTypeDouble" }
    ],
    features: [
      { attributes: { LakeCount: lakeCount, Acres: Round(totalAcres, 2) } }
    ]
  }
);

[LukeGilner1]

Awesome, thanks @KenBuja.  This takes about 1.5 minutes to run, not too bad.

Something strange though.  The totalAcres += Area(lakePart, "acres"); returns 3,815.7 acres,

but changing it to: totalAcres += AreaGeodetic(lakePart, "acres"); correctly returns 1808.9 acres.  (ArcPro shows 1807.9).