topic Big Dataset load time issue in ArcGIS Insights Questions

Big Dataset load time issue

Ed_ — Thu, 28 Apr 2022 13:47:41 GMT

Good day @Scott_Aulen and all ,

Hope all is well, I have a very big dataset that has 700,000 rows yes 700,000 rows with 60 columns. When I initially did some analysis on Insights Desktop with 7 variables (7 columns) and shared the workbook file (a little over 1 GB) with my supervisor, it took around 2 hours just to load the file for the first time.

The workbook file sits on a network drive, but even if loaded from a local drive, it does not make much of a difference.

Now, if I bring in this huge dataset into Insights, first it will take quite a bit of time to import. Then, the saved workbook file will be in GBs, and will take much much more time to load in Insights.

My computer specs is a core i7 10th gen and 32 GB of RAM.

Please note that load time for this dataset is not that much of an issue with PoweBI and RShiny.

But since the preferred tool is Insights, I am need of some kind of an optimization solution to significantly reduce the load time.

Re: Big Dataset load time issue

Scott_Aulen — Wed, 11 May 2022 17:45:54 GMT

Saadullah,

Are you still experiencing this issue? What version are you using?

I'm able to upload a CSV file (~1.5 million records, 1.33 GB) on Insights Desktop in less than a minute.

Scott

Re: Big Dataset load time issue

Ed_ — Tue, 17 May 2022 18:25:44 GMT

Good day Scott,

Thank you for the follow up, I just uploaded (right now it's loading) a zipped GDB which is close to 1.3 GB into an Insights workbook which already has 7 pages.

However, the above dataset is a subset of the original dataset (close to 48 million rows) (CSV) which has a size of 13 GB. I would next try to upload this into that project as well to see if there are any loading issues. Or maybe I will create a new workbook and upload it there to save some load time. So, I don't now if the load time of this big dataset would be the same for uploading in a workbook with 7 pages vs uploading in a new workbook with a single page.

Re: Big Dataset load time issue

Ed_ — Tue, 17 May 2022 19:00:02 GMT

I will add though if Insights supported multiple column/variable selection in a single drop down menu, that would drastically reduce the data size. Because right now, in order to have such a selection, I have to pivot long the dataset which increases the number of rows by multiple folds thus increasing the data size.