Using Insights as a Data Quality reporting tool

[NoelWilson]

I have several technical questions that I have not found a solution for regarding Insights. I have taken and viewed all available training, I have spoken directly with the instruction from the Instructor-led course and there is no resource to find out if what I want to do is even possible.

1. Use, ingest display Data Reviewer reports as a resource for data quality

2. For one of the metrics, the goal is to find all occurrences when a value is blank or null, but when I try to create a query, and I don not manually removed all values but the null, it returns nothing.  The null values are treated as non-values.  Is this a bug?

3. I would like to create a bar chart to show the results of my queries and generate a percentage and show that percentage ON the bar.

4. Create an interface that can be used at an AF Base connected directly to their SDE database server and repeated for all installations.

These are just a few inquires.  I am trying to create  a proof of concept that is repeatable and adaptable.

[Ian_Muehlenhaus]

Hi Noel,

Thanks for reaching out. I'll be honest, I don't have any specific answers for you – yet.

But I want you to know we read these posts and discuss them. So I'll answer what I can and ask someone with more expertise in each of these areas to follow up or check this out. 

1) Could you expand on this first one a little bit for me? I think I know what you mean, but I just want to be clear when I ask about this. 

2) This sounds like something that was never brought to our attention before. There may be an AGOL architecture issue behind this too. I will ask around. However, I just created a video showing an easy way to count/sum <no data> values. I'm hoping this solves your problem, or at least helps you avoid doing everything manually. Please let me know if it helps. If not, I'll pursue the question further. 🙂

https://youtu.be/zGl493flmTc

3) This is a neat idea. I'm not sure we can do that out of the box yet, but I have an idea for a work-around. I'll play with it soon and see if I can get it to work. I'll also mention this in our development idea meetings.

4) I think I know what you mean, but again: are you looking for a way to image the same database connection to automate installs of Insights on many machines?

Thanks again for using the Community! Sorry I don't have more answers right now, but thanks for asking and keep them coming. 

Best,

Ian 

[NoelWilson]

1. From the extension Data Reviewer, you can generate a report that is based on checks that you run in Data Reviewer.  I was wondering if those can be accessed in any way, uploaded to Insights or the information in the report added to a report card.

2. Yes that is how I normally get the null values, BUT if there are 100 values, I currently have to check them all off one by one.  you cannot uncheck ALL and then just select the No DAta value.  Also, technically <Null> is not No data, if it were BLANK it would be No Data.

3.  Looking forward to it

4. When I say installations I am not talking about the installation of the software, it is a term used interchangeably with Air Force Base.  I need to create a workbook, with report cards that can be adapted for a location, and then make that workbook repeatable with a different data source at another location.

[Ian_Muehlenhaus]

A-ha. Now I get it.

1) I don't think ArcGIS Insights syncs with Data Reviewer yet. I'll post this in our ideas board and see if there are any work-arounds out there. Of course, synchronization also depends on the Data Reviewer developers as well. If possible, I would let them know on their discussion board that you would love that feature with ArcGIS Insights. 

2) Wait... My bad! So you can't select <Null> as an option? That probably isn't a bug. It's probably just an oversight, because no one has brought this up before. I will double check and see if it should be something added to the next release later this year. I can't imagine how infuriating that must be!

4) Have you tried exporting a model and a theme? I imagine you have, as you seem like a power user, but if not, you can export a theme (e.g., layout template) and a model (for data processing). You do need to update which data should be used, though, when it changes, unless the file names happen to be the same in each install.

This seems like something a Python script may be able to solve, but that's beyond my scope right now. I may look into it, though, and see if a quick hack isn't possible. Who knows? Python makes miracles happen! 😉

Thanks for your questions. I'll raise these in our ideas board. I'm sorry about the <Null> issue. That will definitely be elevated by me. Hopefully it's an oversight and not an AGOL issue. If that's the case, I imagine we can fix it and save you hours. 

Thanks, Noel. Thanks for your service too! 

Ian

[NoelWilson]

Any more information on the ability to filter "No data" features?  Also, in regards to "No Data" features, if I have a map with location enabled and create a chart  to show the number of features that have the "No Data" value for a specific attribute, when I select that bar in the chart, it will not show me the features on the map.  t works for all other values, but not "No Data".  I need it to be able to do this because I have to figure which Air Force Base has the offending feature.

Also, I have created several report cards for various shapefiles with respective attributes.  I need to be able to share these out to the installation directly, but they will have a different data source.  Is there an easy way to change the data source and not have to recreate the cars and workbook pages from scratch.

[Ian_Muehlenhaus]

Hi Noel, 

I haven't forgotten you. In fact, I've been thinking about this after working hours all week and...

I think I may have figured out a Python-script solution that will do exactly what you want.

I mocked it up and just need to make sure it actually works. Then I need to experiment with exporting it in models, etc. I'll explain my thought process a bit here and post when I know it's functioning right.

------

First things first: I hadn't noticed this before, and maybe you had, but I learned something new:

1. Strings are given the label <No Data> when they are left blank in an imported CSV.

2. Numbers are simply left blank when they are blank.

This doesn't help you per se, but it highlights why sometimes you can select <No Data> and other times you, when you have no values entered or zeros, you can't. 

--------

So yes, you can filter "no data" values from strings fairly easily. They show up as an option when creating predefined or other filters. But that's not your issue, I know... and...

It gets trickier with Numbers, because they don't show up. They are just blank. Or zero is not selectable. So you can't filter the <no data> values out.

Alas, ArcGIS Insights now allows us to write and/or import Python and R scripts! 🙂

In particular, using the Pandas library in Python, we can replace BLANK values in a dataset with a number that is easy to find and export it as a new dataset to our workbook, without impacting the original CSV data. 

Plus, Python scripts can be included as part of any model you export to your different organizations/subunits to use. So theoretically, you can have someone run the same script in the same model every time with different datasets to get the same results. No coding required! 

At least that's the plan... 

The Python script I wrote (and will obviously share, so all someone has to do is drag'n'drop a file from the Table of Contents) works! I've got that down. The model part is next. 

I have a new weekend project! 🙂

Thanks for using ArcGIS Insights, Noel!

I hope to get back to you soon with a solution so you don't have to filter manually ever again! 🙂

[Ian_Muehlenhaus]

Hi @NoelWilson,

Well now I feel daft! But I learned something! 🙂

An even easier solution than using Python scripting?

Use ArcGIS Insights to get a Null Value count!

 

 

You can simply (compared to my last trick!) add a field to the reference table. Then type in "ISNULL" and then select the attribute you want to check. 

Don't forget to rename the new field if you're doing this many times, obviously, or it could get confusing.

Sorry to take you on such a whirlwind tour. But we solved this! Next... to set up a model/template so everyone can do this automatically! 🙂

[NoelWilson]

Okay, I think I figured out a solution to my "Null" issue, I created an
advanced query using the IsNull fx in each table card.



Now I have another problem. How do I create a template of the cards I have
created. I need to be able to share this and change the data source. For
one reason I have to create 3 cards for 19 feature classes. In searching for
that functionality it tells me to choose the share on the workbook page, I
do not see that anywhere. The only share I see is on the Home Page and it
only shares images or data. Is it a user role issue or a license level
issue?

[Ian_Muehlenhaus]

This I can help with! 🙂 I'll upload a solution shortly! Yippee!!! It may not be the exact solution you're looking for, but I think it's what you need. 

Quick follow-up: do your users have Insights Editor access or just viewer status? 

We can make it work either way, but if they have Editor capabilities, it will save you time! 🙂

[NoelWilson]

That has not been determined yet. I am still working on proof of concept
for this specific project. SO far I am finding it is not agile enough to be
a wide are solution for this type of interface, but I will continue to try.