Working With Large Datasets in Insights

[AshleyPiper]

I am having issues with working with large sets of data in insights, particularly with creating a chart/graph. I have 20 or so individual features, each with a few hundred thousand records (data continues to be added to this). When attempting to create a graph, there is an error message telling me to try adding data with <250,000 records. 

I'm not sure if this is a limitation of Insights, but if it is, I am wondering if there might be a better place to display/analyze large sets of data if there is not a workaround for this.

Anyone working with analyzing and displaying large datasets - do you use Insights for this? If not, what do you recommend using instead?

[LauraBecht]

Hi Ashley,
I use large datasets on a regular basis in insights. I find that applying a predefined filter on the dataset helps with displaying the data in charts and maps. Do you need to display the entire dataset, or is there a way you can use a predefined filter.
Regards,
Laura Becht

[AshleyPiper]

Hi,

I do have a predefined filter to only show a certain dataset (water levels recordings) and Insights also defaults to showing a moving average of all of the data. Ideally, I want to show all of it over time (the span of a few years) since it's trending groundwater level data, but for some features that we have years worth of data, it's too much for the graph to handle. 

[KetanDoshi]

Hi Ashley,

Are you using feature/map service from your organization's published services to analysis the data ?

I have learned a hard way that only hosted services works the best for insights to analysis data efficiently.

Defining a relationship in a workbooks also only works on hosted services.

We have more than 600000 parcel data and I have published hosted services with Parcel shapes and CAMA data in simple tables and then make a relationship in insights.

This setup works for us to create all type of charts and maps, though I have to refresh the service daily through a python script 

Predefined filters as suggested here by Laura will also help you get better performance on any object.

[AshleyPiper]

Hi, 

Yes, I am using a published feature service within my organization for this. That might be the issue - I may try publishing as a hosted layer to see if that works better. I've tried the predefined filters, and that still doesn't seem to work for me so I will give that a shot, thanks!

[LindaBeale]

Hi Ashley,

I am wondering if your data is remote, in that it is stored outside of your organization's data store?  This link may be helpful https://doc.arcgis.com/en/insights/latest/administer/configure-insights.htm

There is a limit for remote datasets but you can configure it.

[AshleyPiper · Accepted solution]

Hi,

The data is not remote, though we may eventually want it to be so this will be a great resource for when we eventually make that change - thanks!