News for hackers. Stuff that matters.
Re:When your data doesn’t fit in memory: the basic techniques by itamarst on Monday November 11, 2019 @04:15PM (#21508985)
See above about streaming (chunking is basically the same thing).

mmaping is a form of uncontrolled chunking, where chunk size and location is determined by operating system's filesytem caching policy, readahead heuristics, and the like. As a result, it can sometimes have much worse performance than an explicit chunking strategy, especially if you just treat it as magic.

Or to put it another way, mmaping is helpful but you still need to understand why it might help and how to use it.

Re:When your data doesn’t fit in memory: the basic techniques by faizshah on Tuesday November 12, 2019 @03:46AM (#21512589)
> it can sometimes have much worse performance than an explicit chunking strategy

Can you expand on this?

I haven’t heard this before.

Edit: Found a comprehensive discussion on this https://stackoverflow.com/questions/45972/mmap-vs-reading-bl...