When will Arcade get more useful, esp with FeatureSet?

[DougBrowning]

Still wish there was a dedicated Arcade board.  It is not just AGOL anymore.  Anyway.

I have been using Arcade a ton.  Lots of useful stuff.  Most of my use cases use the FeatureSet functions.  Two biggest uses are grabbing info or a summary between parents and children in relationship classes and aggerating stats of Points in Polygons.  These work great and I can do some cool stuff.

Unfortunately only being able to use FeatureSet in a popup is really limiting what I can do for the user.  While popups are great the times a user will be clicking on each say polygon one by one is pretty slim. 

The problem is you cannot really do anything with the expression except look at it.

Lets say I have a calculation that does an avg of Bare Ground across all points in a polygon.  I want symbolize the polygons say red to green based on the Avg - nope can't.  Ok how about filter the map to show only Avgs under 40% - nope can't.  Ok I will open the attribute table and browse or sort - nope can't do that either.  Ok how about a label that shows good/bad - nope can't do that.  User finally says forget it I will export the data - nope Arcade goes away!

So users come back to me and say this is pretty and all but I cant actually do anything with it.

How are people dealing with this?  Again I love Arcade and its powerful but I am having a hard time finding good workflows to actually do anything with them.

I understand that FeatureSet can get slow and that will cause many of these to be painful.  I have suggested things like adding a where clause to the FeatureSet calls that would speed them up.  I have created a Arc Idea to calc the Arcade values on export.  Labels I think would be ok.  Filter I really thought would work but no FeatureSet is blocked.

Will any of these be opened up to FeatureSet at some point?  Any suggestions or tricks people have used?

Thank for any insight

[jcarlson]

I remember your post about a "where" clause, and discovered something interesting on that point. When you bring in a FeatureSet and then subsequently apply a Filter or Instersects function on it, your browser will actually submit a query to the original feature service, not the in-memory set you've requested.

Say I request a FeatureSet and ask for no fields, no geometry. I can actually still use Filter against any field in the service, and run a spatial intersection against the layer, because those happen on the server side.

But to your larger question of how to actually use Arcade for more advanced things, I agree that its potentially use for things people want, like symbology, seems teasingly out of reach to those particular Arcade profiles.

For us, by far the most interesting and most useful application of Arcade has been in Dashboards, by means of Data Expressions. While it still can't translate to the map, we can still do things there that are impossible anywhere else in the off-the-shelf tools available to us.

So maybe the symbology itself can't change based on an ad-hoc field, but we can use a Data Expression to get the Dashboard to treat our calculated fields as though they were "baked in". Here's an example of using a List with a Data Expression.

Esri's census features have their own "diversity index", but I wanted the Simpson's Index of Diversity, which wasn't in the layer. So I used a Data Expression to create it as a field, and was then able to alter the color on my List using the calculated index.

Not as good as changing the symbology itself, but it's at least something.

Would be curious to see how others answer this question!

[DougBrowning]

Thanks @jcarlson  I already started on the dashboard to do just that.  Having other issues with dashboards though.  When building a FeatureSet in the data expression I am trying to use geometry but it says that Geometry is not a valid function for some reason?  Like umm its right there dude.

I have also noticed what you said on Filter.  I can even say return geo false then load that into a Intersects and somehow it still works.  That means more calls like you said which may explain why its so dang slow.  This seems like it would be super inefficient right?

I may try and go by to ByRelationship it just has so many bugs in it.  Testing it does not seem much faster.  I wonder if it calls them all then filters but that would be silly right.

thanks

[jcarlson]

I actually find that those additional calls make for better performance, because I can get a FeatureSet with, say, 2 fields and no geometry, but then subset that using functions that access any number of fields and geometry. It puts the work on the server, as opposed to some in-memory FeatureSet in my browser. The FeatureSet(s) going to and from the server are comparatively tiny when it's only a couple of fields, as opposed to requesting polylines or polygons[o

Making multiple small requests = lower net I/O = faster execution.

I'd be curious to see the expression you're attempting. I don't seem to run into the same issues.

[DougBrowning]

Answered in the other post but I have some code where I want to make one FeatureSet call with one field then I loop that in a Filter.  If Filter does a new call for every one of the 5,000 items in my loop that has to be slower than using in memory.  On our VPN all these calls actually make a big speed difference.  I do not see why it would have to go back to the server on that.  User clicks popup, it does one call, then a bunch of filters, then display.  Why more calls to the server?

thanks

[jcarlson]

It's interesting, because it doesn't always behave this way. Only when the expression can "see" the original link to the service. If you alter a FeatureSet substantially, additional calls will be made against the in-memory object. And it's only done with certain functions, too. One of the back-end things that I would like to understand better, because knowing when this behavior is or is not exhibited could help users write better expressions. Or better yet, give us some means of controlling this behavior!

If I had to guess, this is probably a case where the developer assumes a particular "standard" use case, builds the tool to facilitate that, and unintentionally makes other uses less efficient. In expressions I write, I want this behavior, but it's very clear that the way your expression evaluates, it is a net loss. And as you state, it doesn't have to go back to the server. There might be workarounds, though...

[jcarlson]

Is the expression in that other post what you're referring to here? You could do something like call Distinct on your FeatureSet. This returns a new FeatureSet, and importantly, once which has its "link" to the original source broken, thus foisting the work to your browser's memory.

You can even sneak your SQL filter into Distinct. This expression (in theory, can't test against the real data), would return a FeatureSet that consists of only the PointIDs of rows matching the filter, along with a single row that has an empty string.

Also, backtick strings in Arcade will respect line breaks, making a long expression much easier to format and keep straight.

var sql = `
CASE
    WHEN
        ResponseType = 'Log an issue'
        AND (Resolved IS NULL OR Resolved = 'No')
    THEN 'PointID'
    ELSE ''
END
`

var unresolved_ids = Distinct(
    input_featureset,
    {
        name: 'unresolved_id',
        expression: sql
    }
)

Another option would be to use Distinct on your input FeatureSet(s) right away. You can use the Schema function to pull out the fields and ensure that all are present in your output.

var fs = some_featureset_object

var flds = Schema(fs)['fields']

var fieldnames = []

for (var f in flds){
    Push(fieldnames, flds[f]['Name'])
}

return Distinct(fs, fieldnames)

This will return an object identical to the input FeatureSet, but with that link severed. See if it improves things for you.

[jcarlson]

And of course, it's always posting here that I realize half a dozen places where I can benefit from this idea! I have a few dashboards where I simply accepted long execution times as a necessary evil, but I'm going to be making some changes today. Many thanks for bringing this up!

[DougBrowning]

Thanks I was planning to post to dig into this today.  I am trying to find the Esri person that always posts but I cant find it.  I wish they could chime in here.

What you are saying may explain why the first code my coworker tried would run forever and never return.

Basically call FeatureSet once and put it in a var - this is were most languages would then use that var in memory.  Then my loop is just a Filter on that in memory set.  But as you point out it looks like each one of these 5,000 will be another call.

tbl2All = FeatureSetByPortalItem(Portal(p),'713e3aaef9674e3493a64347d333b618',10,['PointID','ResponseType','Resolved'],false)
for (var f in tbl) {
    // one call to FeatureSet then loop the filter
    var sql = "PointID = '"+ f.PointID + "' And ResponseType = 'Log an issue' And (Resolved IS NULL Or Resolved = 'No')";
    var tbl2 = Filter(tbl2All,sql);  //so right here it is going back out to the web??
    if (count(tbl) > 0) {
        var v_CountUnresolved = "Yes"
    }
    else {
        var v_CountUnresolved = "No"
    }

This probably explains why when I loaded the values into text instead that was faster since the text string to check is now really in memory.

The other way I wanted to do is to simply load the FeatureSet into an Array.  I am only grabbing one field so it would just be a simple array like [1,2,3].  Then I can loop on that array and it will all be in memory.  But I have found no way to do that.  Array() just makes a array of static values - which is super weird really like what is the use case for that.  I thought Array(FeatureSet) would work but it is not like that.  That is how other languages would work.  Know a way?

Like you said it is hard to tell what IS the most efficient way now.  No doc on this that I can find.  

In Python I got burned on FeatureSet also actually.  I tried it after following the help.  Turns out its the same deal.  Load a URL into a FeatureSet - this is surely in memory - then loop it.  Nope its not in memory and merely a pointer.  This meant I was making thousands of calls to the service.  On slow connections it was brutal.
plotcharF = arcpy.FeatureSet()
plotcharF.load(plotcharURL)

I went back to my old way of FeatureClassToFeatureClass(URL, "in_memory", name) and the script went from hours to 3 minutes!  Again no mention of this in the help that I saw.

I am with you I would love to know the details so I can write good code.  Having filter go back out to the service is weird for sure.  Like why bother asked me for a field list in FeatureSet then?

thanks for working though this

[jcarlson]

I believe you're looking for @MikeMillerGIS ? I think he's involved in Arcade on the Esri side of things.

Okay, I initially thought that converting your stuff through text wasn't a great idea, but I'm seeing the use for it. Trouble is, there's still not a nice and easy way to do that without using a loop to build the string first.

What Arcade could really use is a "to_array" or "to_dict" function that we could call on a FeatureSet or something...

Attempting even more convoluted methods of severing that tie, and I am not certain I can actually do it. I may simply be thinking of FeatureSets I've created "from scratch" for testing purposes, or intermediate FeatureSets created by FeatureSet(Text(some_dict)), but then used in a subsequent function, rather than being the output itself. Arcade seems to cobble together the various FeatureSetBy... functions into whatever consolidated queries it can.

That does help understand that a subsequent filter is literally the same as supplying "out fields" to the initial FeatureSet function, as the browser only sends one query, with the filter applied then.

A bit opaque for my taste, but it's the only game in town.