Dashboard Data Export not correct when using spatial filters

[AMoyers]

Hi all, I have a dashboard that displays two datasets: one is a dataset with about 40,000 points and another is a polygon dataset that contains about 65 circular 14-mile buffers.  The dashboard has a series of filters that filter by attribute and then there's one "spatial filter" to narrow the points to those in a chosen buffer.  There's a "list" widget, a pie chart widget and then an indicator widget that populates based on the filters to display information about the points currently on the map.  Using the new data export functionality added in the September ArcGIS Online update the data does not line up between export and what's displayed in the dashboard, either on the map or in the widgets when we use the buffer spatial filter.

This issue appears to happen only when the spatial filter is applied.  For example I tried a test where the dashboard and map displayed 173 unique points but there were 178 unique entries in the export.  I determined that the extra 5 points were all outside of the spatial filter, no intersection and the largest distance was at least 2.5 miles away.  Is this a bug or has someone else run into this?

I'm frustrated because of the level of anticipation on this functionality, if I can't get this figured out we are probably going to abandon the dashboard.  I work for a food bank, we are using this to determine clients for home delivery by food delivery services -- the export functionality is necessary to take the next step and make it an actionable interface.

Any ideas?  Thanks in advance for any help--

[DavidNyenhuis1]

Hi @AMoyers, 

Sorry to hear about this problem. We are looking into it.

If you are able to share them, it would be helpful to have access to the dashboard and/or layer(s). You could do that here or by logging the bug with Esri Support (which would also help us track it better and get additional details that may be necessary). If the data cannot be shared, Support can work to replicate the issue with other data and get it sent to the right team.

Thanks,

Dave

[AMoyers]

@DavidNyenhuis1 thank you for responding -- I just reported a bug at https://support.esri.com/en/report-bug.  It looks like I don't have access to support chat or tracking a case within my organization right now.  I might be able to generalize the data and churn up an example to replicate.

[DavidNyenhuis1]

@AMoyers, thank you for reporting this. It was found to be working as designed: when using spatial filters based on a polygon, the CSV will include features intersecting the envelope (i.e., rectangle around it). We plan to work toward improving this in the future. 

If exact results are required, as a workaround we suggest using attribute-based filtering. Having common identifier fields across layers will allow you to set this up. 

All the best,

Dave

[AMoyers]

@DavidNyenhuis1 can you point me to documentation that discusses this, for future reference?  Thank you--

[DavidNyenhuis1]

Please see this post for more details.

[AMoyers]

@DavidNyenhuis1 is there any timeline for Esri looking at the export functionality with spatial filters?  I have been researching and trying to work around this issue since October.  As a workaround we generated bounding boxes around the buffers.  Theoretically I would hope this would match the envelope and it is much closer to the data that's showing up in other widgets like the indicator and list widget from which it exports but we are still getting extra results.  Are there any plans to document the way this functionality works other than in a forum post?

The way this has been implemented, where the CSV output does not match what's displayed in the widget, seems intentional.  After all, it's the same data that AGOL is already populating into the widget.

[NeilWebber]

@AMoyers, @DavidNyenhuis1 

Hi both, there seems to be some in-house discrepancy at ESRI on whether this issue is a feature or a bug. My organisation raised this same issue as a ticket last year and the final conclusion (4 Nov 21) was as follows:

"...I have received word back from Esri Inc in relation to this matter. They have logged this as a new defect on our system known as BUG-000144262 - Downloaded CSV from spatial filter in ArcGIS Online Dashboard shows wrong output records.

As this BUG is new, I'm afraid there is no information available at the moment regarding a date/version a fix will be released, however, I would like to reassure you that the Operation team and the developers are trying their best to fix this bug. I have attached the BUG to this case so the customer can keep a track of its status via My ESRI. Please allow some time for the ESRI Inc Development team to review the BUG and provide an update. The severity of the BUG is listed as "Medium" so it is likely to be reviewed sooner rather than later..."

This response seems clear that the export of data contained within an envelope rather than the feature itself is not by design; and why would it be? - there's no reason I can think of to build it like that - it's counter-intuitive to me, and feels like a quick route to unexpectedly erroneous data. You actually have to work quite hard to get a spatial selection on an envelope rather than a feature in ArcMap/Pro for instance.  The fact that various dashboard metric elements (lists, indicators, gauges) correctly recognise the spatial filter, but the output csv doesn't, is an indication in itself that something is not right.

The 'envelope' behaviour is not documented in the support guide, which still maintains that "...downloaded data respects all filters applied to both the data source and the element, including layer filters, web map filters, and filters from actions and selectors..." At the very least, while the bug exists, it would be helpful to amend this to draw attention to what will, for most users, be unexpected behaviour - it took me some while to figure out what was happening to give me what I considered to be an incorrect output.

I've abandoned this particular tool for now, which is a shame because it could be spectacularly useful, but I really can't think of any realistic use-case for it in its current state.

Dave - could we get an update on where BUG-000144262 is at the moment?

regards, Neil

[AMoyers]

@NeilWebber Thank you for posting back to this with what you've been able to determine from Esri.  In the process of trying to figure out where the extra points were coming from in the exports I did some follow up tests with the exports.  Because the density of points was so high in some of the areas where we were using the spatial filter, it was sometimes resulting in several thousand extra results in the CSV output.  The data consistently lined up normally to the expected geometry in all of the widgets but the export appeared to be consistently applying some type of bounding box geometry to the output.  I re-created feature layers from the outputs with thousands of extra points and all of these points always lay outside of the circle but within a square geometry around the circle -- i.e. they existed in NE/SE/SW/NW areas around the circles.

I agree with your point completely that doing it based on the envelope seems even more complicated than just doing it as the true geometry.  I originally submitted a ticket on this but the time required to anonymize and set up examples for Esri to replicate this problem was going to take me hours and without it being clear whether or not it was intentioned functionality the investment in time was also unclear.

[DavidNyenhuis1]

Hi  @AMoyers @NeilWebber ,

Thanks for following up. Thank you for your patience and sorry about the confusion about the classification of the issue. 

We have updated the API to now take the actual geometry into account when doing the spatial filter on a CSV export. So, when you apply a spatial (polygon) filter action to a list of features, then export the source data, you will get the same number of rows in your CSV.

This update is planned to roll out with next week's ArcGIS Online release. Please let me know if you have any other questions or concerns with this implementation.