Working With Large Datasets in Insights

778
6
Jump to solution
07-20-2021 09:57 AM
AshleyPiper
New Contributor II

I am having issues with working with large sets of data in insights, particularly with creating a chart/graph. I have 20 or so individual features, each with a few hundred thousand records (data continues to be added to this). When attempting to create a graph, there is an error message telling me to try adding data with <250,000 records. 

I'm not sure if this is a limitation of Insights, but if it is, I am wondering if there might be a better place to display/analyze large sets of data if there is not a workaround for this.

Anyone working with analyzing and displaying large datasets - do you use Insights for this? If not, what do you recommend using instead?

1 Solution

Accepted Solutions
AshleyPiper
New Contributor II

Hi,

The data is not remote, though we may eventually want it to be so this will be a great resource for when we eventually make that change - thanks!

View solution in original post

0 Kudos
6 Replies
LauraBecht
New Contributor III
Hi Ashley,
I use large datasets on a regular basis in insights. I find that applying a predefined filter on the dataset helps with displaying the data in charts and maps. Do you need to display the entire dataset, or is there a way you can use a predefined filter.
Regards,
Laura Becht
AshleyPiper
New Contributor II

Hi,

I do have a predefined filter to only show a certain dataset (water levels recordings) and Insights also defaults to showing a moving average of all of the data. Ideally, I want to show all of it over time (the span of a few years) since it's trending groundwater level data, but for some features that we have years worth of data, it's too much for the graph to handle. 

0 Kudos
KetanDoshi
New Contributor III

Hi Ashley,

Are you using feature/map service from your organization's published services to analysis the data ?

I have learned a hard way that only hosted services works the best for insights to analysis data efficiently.

Defining a relationship in a workbooks also only works on hosted services.

We have more than 600000 parcel data and I have published hosted services with Parcel shapes and CAMA data in simple tables and then make a relationship in insights.

This setup works for us to create all type of charts and maps, though I have to refresh the service daily through a python script 

Predefined filters as suggested here by Laura will also help you get better performance on any object.

AshleyPiper
New Contributor II

Hi, 

Yes, I am using a published feature service within my organization for this. That might be the issue - I may try publishing as a hosted layer to see if that works better. I've tried the predefined filters, and that still doesn't seem to work for me so I will give that a shot, thanks!

0 Kudos
LindaBeale
Esri Contributor

Hi Ashley,

I am wondering if your data is remote, in that it is stored outside of your organization's data store?  This link may be helpful https://doc.arcgis.com/en/insights/latest/administer/configure-insights.htm

There is a limit for remote datasets but you can configure it.

AshleyPiper
New Contributor II

Hi,

The data is not remote, though we may eventually want it to be so this will be a great resource for when we eventually make that change - thanks!

0 Kudos