Re:I wrote one of the fastest DataFrame libraries
by hermitcrab on Sunday March 14, 2021 @05:57AM
(#26453996)
How is that different to virtual memory?
Re:I wrote one of the fastest DataFrame libraries
by nijave on Sunday March 14, 2021 @11:17PM
(#26462266)
In memory representation also tends to have some data structure overhead so 5GB compressed -> 50GB uncompressed -> 100-250GB as a data structure (seems 2-5x is pretty normal). It starts out seeming pretty innocuous but quickly explodes. In addition, some things like Drill can do some automatic indexing/metadata recording which can reduce the amount of data it needs to access
Re:I wrote one of the fastest DataFrame libraries
by dTal on Sunday March 14, 2021 @08:33AM
(#26454787)
Memory mapping lazily loads the file, completes immediately, and scales arbitrarily; using disk-backed virtual memory would require the entire file to be read from disk, written back out to disk, and then read in from disk again on access; it would also require swap to be set up at the OS level, and the amount of swap set up puts a hard limit on the size of the file.
Re:I wrote one of the fastest DataFrame libraries
by wtallis on Sunday March 14, 2021 @08:55AM
(#26454981)
You've described the difference between using mmap vs relying on the operating system's swap mechanism. But neither of those is quite the same as having an application that's aware of its memory usage and explicitly manages what it keeps in RAM. Using mmap may be useful for achieving that, but mmap on its own still leaves most of the management up to the OS.
Re:I wrote one of the fastest DataFrame libraries
by hermitcrab on Sunday March 14, 2021 @10:56AM
(#26455925)
Thanks for clearing that up!
Particularly useful for when you’ve got a bunch of data that doesn’t fit in memory, but setting up a whole cluster is not worth the overhead (operationally or otherwise) and/or if you’ve already got a processing pipeline written in a language/framework and you can’t/don’t-want to go through rewriting it for something distributed.