Re:I wrote one of the fastest DataFrame libraries
by nijave on Sunday March 14, 2021 @11:17PM
(#26462266)
In memory representation also tends to have some data structure overhead so 5GB compressed -> 50GB uncompressed -> 100-250GB as a data structure (seems 2-5x is pretty normal). It starts out seeming pretty innocuous but quickly explodes. In addition, some things like Drill can do some automatic indexing/metadata recording which can reduce the amount of data it needs to access
[ Parent
| on HN ]
There are no comments on this comment yet.