csny490

Large Dictionary Compression?

Discussion created by csny490 on May 23, 2012
Latest reply on Feb 22, 2013 by csny490
I have a simple dictionary like this:

exampleDict[123444556] = (1785,2234544,3545456, 165765.47654)

where all the keys are intgers and the values are either integers or floats.

My issue is that I have the need to store/access about 20 million keys at a time, and I am running out of 32-bit memory. I'd rather do this in 32-bit Python as I need (or would like) access to arcpy for its FGDB table reading/writting abilities.

Anyone know of a way to somehow "compress" keys and/or values in a dictionary? I'm looking into the binascii module, and I see lots of methods to compress strings, but not ints or floats. Maybe you can't meaningfully compress these since they are already quite numeric?

Anyone ever do something like this?

Outcomes