News for hackers. Stuff that matters.

A comment on Big data is dead (2023). Parent comment

Re:Big data is dead (2023) by ninkendo on Monday May 27, 2024 @06:43PM (#40495494)
Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram.

It’ll be slower, perhaps by a lot, but most “big data” stuff is already so god damned slow that mmap probably still beats it, while being immeasurably simpler and cheaper.

[ Parent | on HN ]
Re:Big data is dead (2023) by marginalia_nu on Monday May 27, 2024 @08:14PM (#40496079)
Really depends on the shape of the data. mmap can be suboptimal in many cases.

For CSV it flat out doesn't matter what you do since the format is so inefficient and needs to be read start to finish, but something like parquet probably benefits from explicit read syscalls, since it's block based and highly structured, where you can predict the read patterns much better than the kernel can.

[ Parent | on HN ]