Re:Big data is dead (2023)
by marginalia_nu on Monday May 27, 2024 @08:20AM
(#40490093)
To be fair, mmap doesn't put your data in RAM, it presents it as though it was in RAM and has the OS deal with whether or not it actually is.
[ Parent
| on HN ]
Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram.
It’ll be slower, perhaps by a lot, but most “big data” stuff is already so god damned slow that mmap probably still beats it, while being immeasurably simpler and cheaper.
Re:Big data is dead (2023)
by marginalia_nu on Monday May 27, 2024 @08:14PM
(#40496079)
Really depends on the shape of the data. mmap can be suboptimal in many cases.
For CSV it flat out doesn't matter what you do since the format is so inefficient and needs to be read start to finish, but something like parquet probably benefits from explicit read syscalls, since it's block based and highly structured, where you can predict the read patterns much better than the kernel can.