Data size limitation

[Ed_]

Good day all,

Is there a data size upload limitation for Insights desktop? I have a dataset with 40,349,178 rows and it's close to 8 GB. 

[Ed_]

Actually it's failing to upload, please note the dataset is a `.gdb` `zip` file. 

[KateMaclachlan]

Hi Saadullah, 
In order to upload a .gdb file you must do this from the datasets tab on the Homepage. You cannot upload .gdb(even if its zipped) in the Add to page dialogue within the workbook at this time - hopefully in the future! 

Once you have uploaded the dataset into the homepage it will appear in your Add to page dialogue under "Local content"

Hopefully this helps!

Kate

[Ed_]

So there's no datasize limitations in Insights?

Moreover, let's say I have a workbook with 6 pages each with big datasets, is there an upward limit to the datasize that Insights can handle? And should adding new pages slow down Insights while working with big dataset?

[KateMaclachlan · Accepted solution]

We dont have any defined limit at this time for datasize in Insights desktop. However there can be issues with performance and certain actions as data size increases. The larger your workbook - the more pages and the more datasets you have has potential to slow things down. We are striving to work towards continued improvements with performance and large data. 

I was just able to upload a .gdb which has 38M records and is 2.6GB in compressed format. (though it did take some time to finish). 
It's possible that you have run into a bug  - be that datasize or an issue within the configuration of the gdb itself. If possible it would be great if you could log a bug with tech support - including the data would be helpful or the schema so we can try and reproduce this issue and get it solved. 

[Ed_]

Moreover, I had to delete the temp files in the windows temp folder to free up some space in my local C drive, and now I am noticing an increase in the load time. The 1.3 GB zipped GDB uploaded rather quickly but in terms of loading it has taken more an hour now. The GDB is currently loading into a workbook that already has 5 pages so far.

[Ed_]

Nope, it also fails to upload this way.

[Ed_]

I will add though if Insights supported multiple column/variable selection in a single drop down menu, that would drastically reduce the data size. Because right now, in order to have such a selection, I have to pivot long the dataset which increases the number of rows by multiple folds thus increasing the data size.

[Ed_]

The data was uploaded from a shared network drive, so I don't know whether uploading it from the local C drive would reduce the load time or not?

[Scott_Aulen]

@Ed_,

Uploading locally should help performance but as @KateMaclachlan suggested there may be something else going on with the GDB itself.  I would also encourage you to log this with Customer Service so that someone can examine this specific issue further.