News for hackers. Stuff that matters.
Re:Big data is dead (2023) by marginalia_nu on Monday May 27, 2024 @08:14PM (#40496079)
Really depends on the shape of the data. mmap can be suboptimal in many cases.

For CSV it flat out doesn't matter what you do since the format is so inefficient and needs to be read start to finish, but something like parquet probably benefits from explicit read syscalls, since it's block based and highly structured, where you can predict the read patterns much better than the kernel can.

There are no comments on this comment yet.