News for hackers. Stuff that matters.
Re:Are You Sure You Want to Use MMAP in Your Database Manage... by jandrewrogers on Sunday July 02, 2023 @06:20PM (#36566403)
I think the real argument is more nuanced. Where you see mmap() fail badly on Linux, even for read-only workloads, is under a few specific conditions: very large storage volumes, highly concurrent access, non-trivial access patterns (e.g. high-dimensionality access methods). Most people do not operate data models under these conditions, but if you do then you can achieve large integer factor gains in throughput by not using mmap().

Interestingly, most of the reason for these problems has to do with theoretical limitations of cache replacement algorithms as drivers of I/O scheduling. There are alternative approaches to scheduling I/O that work much better in these cases but mmap() can’t express them, so in those cases bypassing mmap() offers large gains.

Re:Are You Sure You Want to Use MMAP in Your Database Manage... by pclmulqdq on Sunday July 02, 2023 @08:03PM (#36567136)
GP wrote a key-value store called LMDB that is constrained to a single writer, and often used for small databases that fit entirely in memory but need to persist to disk. There's a whole different world for more scalable databases.
Re:Are You Sure You Want to Use MMAP in Your Database Manage... by hyc_symas on Sunday July 02, 2023 @09:18PM (#36567590)
"fit entirely in memory" is not a requirement. LMDB is not a main-memory database, it is an on-disk database that uses memory mapping.
Re:Are You Sure You Want to Use MMAP in Your Database Manage... by LAC-Tech on Sunday July 02, 2023 @08:33PM (#36567332)
Can you explain "high-dimensionality access methods" to me? (Or if it's too big for an HN comment, maybe recommend a paper).
Re:Are You Sure You Want to Use MMAP in Your Database Manage... by heisjustsosmart on Wednesday July 05, 2023 @01:26PM (#36603716)
This guy talks a lot of crap. See his website for examples, and don't waste your time with him

<<<There is one significant drawback that should not be understated. Algorithm design using topology manipulation can be enormously challenging to reason about. You are often taking a conceptually simple algorithm, like a nested loop or hash join, and replacing it with a much more efficient algorithm involving the non-trivial manipulation of complex high-dimensionality constraint spaces that effect the same result. Routinely reasoning about complex object relationships in greater than three dimensions, and constructing correct parallel algorithms that exploit them, becomes easier but never easy.>>>

http://www.jandrewrogers.com/2015/10/08/spacecurve/

Re:Are You Sure You Want to Use MMAP in Your Database Manage... by ilyt on Sunday July 02, 2023 @09:33PM (#36567663)
I'd imagine same kind of worst case access would also be a problem doing IO the "classical" way