News for hackers. Stuff that matters.
Re:I wrote one of the fastest DataFrame libraries by nijave on Sunday March 14, 2021 @11:17PM (#26462266)
In memory representation also tends to have some data structure overhead so 5GB compressed -> 50GB uncompressed -> 100-250GB as a data structure (seems 2-5x is pretty normal). It starts out seeming pretty innocuous but quickly explodes. In addition, some things like Drill can do some automatic indexing/metadata recording which can reduce the amount of data it needs to access

There are no comments on this comment yet.