topic Re: When will Arcade get more useful, esp with FeatureSet? in ArcGIS Online Questions

When will Arcade get more useful, esp with FeatureSet?

DougBrowning — Wed, 19 Oct 2022 14:18:14 GMT

Still wish there was a dedicated Arcade board. It is not just AGOL anymore. Anyway.

I have been using Arcade a ton. Lots of useful stuff. Most of my use cases use the FeatureSet functions. Two biggest uses are grabbing info or a summary between parents and children in relationship classes and aggerating stats of Points in Polygons. These work great and I can do some cool stuff.

Unfortunately only being able to use FeatureSet in a popup is really limiting what I can do for the user. While popups are great the times a user will be clicking on each say polygon one by one is pretty slim.

The problem is you cannot really do anything with the expression except look at it.

Lets say I have a calculation that does an avg of Bare Ground across all points in a polygon. I want symbolize the polygons say red to green based on the Avg - nope can't. Ok how about filter the map to show only Avgs under 40% - nope can't. Ok I will open the attribute table and browse or sort - nope can't do that either. Ok how about a label that shows good/bad - nope can't do that. User finally says forget it I will export the data - nope Arcade goes away!

So users come back to me and say this is pretty and all but I cant actually do anything with it.

How are people dealing with this? Again I love Arcade and its powerful but I am having a hard time finding good workflows to actually do anything with them.

I understand that FeatureSet can get slow and that will cause many of these to be painful. I have suggested things like adding a where clause to the FeatureSet calls that would speed them up. I have created a Arc Idea to calc the Arcade values on export. Labels I think would be ok. Filter I really thought would work but no FeatureSet is blocked.

Will any of these be opened up to FeatureSet at some point? Any suggestions or tricks people have used?

Thank for any insight

Re: When will Arcade get more useful, esp with FeatureSet?

jcarlson — Wed, 19 Oct 2022 15:28:44 GMT

I remember your post about a "where" clause, and discovered something interesting on that point. When you bring in a FeatureSet and then subsequently apply a Filter or Instersects function on it, your browser will actually submit a query to the original feature service, not the in-memory set you've requested.

Say I request a FeatureSet and ask for no fields, no geometry. I can actually still use Filter against any field in the service, and run a spatial intersection against the layer, because those happen on the server side.

But to your larger question of how to actually use Arcade for more advanced things, I agree that its potentially use for things people want, like symbology, seems teasingly out of reach to those particular Arcade profiles.

For us, by far the most interesting and most useful application of Arcade has been in Dashboards, by means of Data Expressions. While it still can't translate to the map, we can still do things there that are impossible anywhere else in the off-the-shelf tools available to us.

So maybe the symbology itself can't change based on an ad-hoc field, but we can use a Data Expression to get the Dashboard to treat our calculated fields as though they were "baked in". Here's an example of using a List with a Data Expression.

Esri's census features have their own "diversity index", but I wanted the Simpson's Index of Diversity, which wasn't in the layer. So I used a Data Expression to create it as a field, and was then able to alter the color on my List using the calculated index.

Not as good as changing the symbology itself, but it's at least something.

Would be curious to see how others answer this question!

Re: When will Arcade get more useful, esp with FeatureSet?

DougBrowning — Wed, 19 Oct 2022 21:06:46 GMT

Thanks @jcarlson I already started on the dashboard to do just that. Having other issues with dashboards though. When building a FeatureSet in the data expression I am trying to use geometry but it says that Geometry is not a valid function for some reason? Like umm its right there dude.

I have also noticed what you said on Filter. I can even say return geo false then load that into a Intersects and somehow it still works. That means more calls like you said which may explain why its so dang slow. This seems like it would be super inefficient right?

I may try and go by to ByRelationship it just has so many bugs in it. Testing it does not seem much faster. I wonder if it calls them all then filters but that would be silly right.

thanks

Re: When will Arcade get more useful, esp with FeatureSet?

jcarlson — Wed, 19 Oct 2022 21:44:14 GMT

I actually find that those additional calls make for better performance, because I can get a FeatureSet with, say, 2 fields and no geometry, but then subset that using functions that access any number of fields and geometry. It puts the work on the server, as opposed to some in-memory FeatureSet in my browser. The FeatureSet(s) going to and from the server are comparatively tiny when it's only a couple of fields, as opposed to requesting polylines or polygons[o

Making multiple small requests = lower net I/O = faster execution.

I'd be curious to see the expression you're attempting. I don't seem to run into the same issues.

Re: When will Arcade get more useful, esp with FeatureSet?

AlfredBaldenweck — Wed, 19 Oct 2022 21:48:51 GMT

I've been feeling this way for a bit, too.

Even just in a normal Pro session, there are tons of things that , like Josh says, are teasingly out of reach.

Particularly, as you say, using FeatureSet outside of popups. We have all these cool things that could work (I wanted to symbolize by whether or not some points were within 3 miles of some turbines), but most profiles are completely agnostic to things other than the feature class itself.

Arcade is great at getting things dynamically. I'd love to be able to use it for quick and dirty visualization without having to make a new layer or a new feature class, but right now it just can't do what I want. I love it for field calculations and pop-ups, but visualization needs a lot more work.

Another use case I noticed last week was this post: Re: Symbolize top 10 attributes by count dynamical... - Esri Community

Why don't we have a $view global that's not just for view scale? Let us adjust symbology by what's in the view, a la DRA for raster datasets.

I second the Arcade Board idea; I'm never sure where to ask questions for it.

Re: When will Arcade get more useful, esp with FeatureSet?

DougBrowning — Wed, 19 Oct 2022 23:31:07 GMT

Answered in the other post but I have some code where I want to make one FeatureSet call with one field then I loop that in a Filter. If Filter does a new call for every one of the 5,000 items in my loop that has to be slower than using in memory. On our VPN all these calls actually make a big speed difference. I do not see why it would have to go back to the server on that. User clicks popup, it does one call, then a bunch of filters, then display. Why more calls to the server?

thanks

Re: When will Arcade get more useful, esp with FeatureSet?

JohannesLindner — Thu, 20 Oct 2022 07:07:49 GMT

In regards to an Arcade community, please lend your support to this idea: Arcade Community - Esri Community

In regards to making featuresets available in the labeling and visualization profile: This would be great, but it's probably not coming any time soon. Post from February 2022:

There are no plans to extend the labeling or visualization profiles to include FeatureSet functionality for performance reasons. Label and visualization expressions are executed on a per-feature basis and a feature set query executed per-feature would slow down draw performance.
For this use-case the recommendation is to use a calculate attribute rule where the script is executed at data creation or update time rather than once per draw loop. FeatureSetByName can be used to perform FeatureSet lookups with any other dataset in the same workspace. Cross-database cases are not currently possible but an idea to support that for future workflows could be submitted for conversation.

Re: When will Arcade get more useful, esp with FeatureSet?

jcarlson — Thu, 20 Oct 2022 13:19:25 GMT

It's interesting, because it doesn't always behave this way. Only when the expression can "see" the original link to the service. If you alter a FeatureSet substantially, additional calls will be made against the in-memory object. And it's only done with certain functions, too. One of the back-end things that I would like to understand better, because knowing when this behavior is or is not exhibited could help users write better expressions. Or better yet, give us some means of controlling this behavior!

If I had to guess, this is probably a case where the developer assumes a particular "standard" use case, builds the tool to facilitate that, and unintentionally makes other uses less efficient. In expressions I write, I want this behavior, but it's very clear that the way your expression evaluates, it is a net loss. And as you state, it doesn't have to go back to the server. There might be workarounds, though...

Re: When will Arcade get more useful, esp with FeatureSet?

jcarlson — Thu, 20 Oct 2022 13:45:27 GMT

Is the expression in that other post what you're referring to here? You could do something like call Distinct on your FeatureSet. This returns a new FeatureSet, and importantly, once which has its "link" to the original source broken, thus foisting the work to your browser's memory.

You can even sneak your SQL filter into Distinct. This expression (in theory, can't test against the real data), would return a FeatureSet that consists of only the PointIDs of rows matching the filter, along with a single row that has an empty string.

Also, backtick strings in Arcade will respect line breaks, making a long expression much easier to format and keep straight.

var sql = ` CASE WHEN ResponseType = 'Log an issue' AND (Resolved IS NULL OR Resolved = 'No') THEN 'PointID' ELSE '' END ` var unresolved_ids = Distinct( input_featureset, { name: 'unresolved_id', expression: sql } )

Another option would be to use Distinct on your input FeatureSet(s) right away. You can use the Schema function to pull out the fields and ensure that all are present in your output.

var fs = some_featureset_object var flds = Schema(fs)['fields'] var fieldnames = [] for (var f in flds){ Push(fieldnames, flds[f]['Name']) } return Distinct(fs, fieldnames)

This will return an object identical to the input FeatureSet, but with that link severed. See if it improves things for you.

Re: When will Arcade get more useful, esp with FeatureSet?

jcarlson — Thu, 20 Oct 2022 13:47:56 GMT

And of course, it's always posting here that I realize half a dozen places where I can benefit from this idea! I have a few dashboards where I simply accepted long execution times as a necessary evil, but I'm going to be making some changes today. Many thanks for bringing this up!

Re: When will Arcade get more useful, esp with FeatureSet?

DougBrowning — Thu, 20 Oct 2022 14:26:57 GMT

Thanks I was planning to post to dig into this today. I am trying to find the Esri person that always posts but I cant find it. I wish they could chime in here.

What you are saying may explain why the first code my coworker tried would run forever and never return.

Basically call FeatureSet once and put it in a var - this is were most languages would then use that var in memory. Then my loop is just a Filter on that in memory set. But as you point out it looks like each one of these 5,000 will be another call.

tbl2All = FeatureSetByPortalItem(Portal(p),'713e3aaef9674e3493a64347d333b618',10,['PointID','ResponseType','Resolved'],false)
for (var f in tbl) {
    // one call to FeatureSet then loop the filter
    var sql = "PointID = '"+ f.PointID + "' And ResponseType = 'Log an issue' And (Resolved IS NULL Or Resolved = 'No')";
    var tbl2 = Filter(tbl2All,sql); //so right here it is going back out to the web??
    if (count(tbl) > 0) {
        var v_CountUnresolved = "Yes"
    }
    else {
        var v_CountUnresolved = "No"
    }

This probably explains why when I loaded the values into text instead that was faster since the text string to check is now really in memory.

The other way I wanted to do is to simply load the FeatureSet into an Array. I am only grabbing one field so it would just be a simple array like [1,2,3]. Then I can loop on that array and it will all be in memory. But I have found no way to do that. Array() just makes a array of static values - which is super weird really like what is the use case for that. I thought Array(FeatureSet) would work but it is not like that. That is how other languages would work. Know a way?

Like you said it is hard to tell what IS the most efficient way now. No doc on this that I can find.

In Python I got burned on FeatureSet also actually. I tried it after following the help. Turns out its the same deal. Load a URL into a FeatureSet - this is surely in memory - then loop it. Nope its not in memory and merely a pointer. This meant I was making thousands of calls to the service. On slow connections it was brutal.
plotcharF = arcpy.FeatureSet()
plotcharF.load(plotcharURL)

I went back to my old way of FeatureClassToFeatureClass(URL, "in_memory", name) and the script went from hours to 3 minutes! Again no mention of this in the help that I saw.

I am with you I would love to know the details so I can write good code. Having filter go back out to the service is weird for sure. Like why bother asked me for a field list in FeatureSet then?

thanks for working though this

Re: When will Arcade get more useful, esp with FeatureSet?

jcarlson — Thu, 20 Oct 2022 14:30:47 GMT

Belay that. Link remains, somehow. I tried getting the distinct and subsequently filtering, and the request to the server included the grouping and the filtering in the same request. Very curious...

Re: When will Arcade get more useful, esp with FeatureSet?

DougBrowning — Thu, 20 Oct 2022 14:47:45 GMT

We have the dashboard now with a data expression as you posted. It turns rows red if a matching error report is found in the Error form layer. And now with the Geometry(f) part right it will even change on zoom which is sweet. thanks on that.

Do you know how to get it to use the same symbology as the map in the list? It works when its a layer but when it goes to data expression it changes all the symbols to the same symbol. I think it may be the no value symbol from the map.

In the properties all I see is symbol on/off.

Is there a way to say hey use the symbology from over on the map? It is that or we have to recreate it in Ops using a big if statement I guess.

thanks

Re: When will Arcade get more useful, esp with FeatureSet?

jcarlson — Thu, 20 Oct 2022 14:48:47 GMT

I believe you're looking for @MikeMillerGIS ? I think he's involved in Arcade on the Esri side of things.

Okay, I initially thought that converting your stuff through text wasn't a great idea, but I'm seeing the use for it. Trouble is, there's still not a nice and easy way to do that without using a loop to build the string first.

What Arcade could really use is a "to_array" or "to_dict" function that we could call on a FeatureSet or something...

Attempting even more convoluted methods of severing that tie, and I am not certain I can actually do it. I may simply be thinking of FeatureSets I've created "from scratch" for testing purposes, or intermediate FeatureSets created by FeatureSet(Text(some_dict)), but then used in a subsequent function, rather than being the output itself. Arcade seems to cobble together the various FeatureSetBy... functions into whatever consolidated queries it can.

That does help understand that a subsequent filter is literally the same as supplying "out fields" to the initial FeatureSet function, as the browser only sends one query, with the filter applied then.

A bit opaque for my taste, but it's the only game in town.

Re: When will Arcade get more useful, esp with FeatureSet?

jcarlson — Thu, 20 Oct 2022 14:55:56 GMT

Yeah, I don't know if the FeatureSet comes in w/ default symbology. According to the REST docs, a Feature can come in with a Symbol property, but I don't recall ever having seen that in Arcade...

A workaround I've used is to use SVG elements and pipe in attributes to them. Like here, I have a list of features and I wanted graduated symbols in the List.

I had to use Advanced Formatting to scale the attribute to a pixel size that made sense, but I was satisfied with the result. Though a bit tedious, you could find SVG versions of the symbols you like, then pipe those into an expression with When(class_a, "svg string", class_b, "other svg string"...)

Re: When will Arcade get more useful, esp with FeatureSet?

DougBrowning — Thu, 20 Oct 2022 15:43:36 GMT

Most of my stuff is using FeatureSet and intersects with a Polygon and points. Hard to tell if attribute rules can handle that yet. Also these go across databases for sure. Some I use Esri layers even. Like summarizes our data points inside HUCs from Living Atlas. So still wont work it sounds like from your text. But for now I am AGOL which has no attribute rules yet anyway.

Bummer. I still think if they could speed up FeatureSet it could work.

thanks

Re: When will Arcade get more useful, esp with FeatureSet?

DougBrowning — Wed, 26 Oct 2022 16:44:48 GMT

@jcarlson Have you noticed if using FeatureSetByRelationshipName is faster? In theory you would think it does the query by key server side and just returns the one record I need.

I was skipping FeatureSetByRelationshipName since it had bugs but seems to work on my data.

Should be faster than this right?

var sql = "PlotKey = '" + $feature.PlotKey + "'";
var tbl = Filter(FeatureSetByName($map,"Soil Stability"), sql);

I tested but have not seem much difference yet.

thanks