News for hackers. Stuff that matters.
Re:Why MMAP in llama.cpp hides true memory usage by quotemstr on Monday April 03, 2023 @02:03PM (#35428534)
Buggy? In what way?
Re:Why MMAP in llama.cpp hides true memory usage by masklinn on Monday April 03, 2023 @02:26PM (#35428917)
The "But there are also disadvantages" of the linked pages provides some of the issues with mmap. This also matches burntsushi's experience with mmap in ripgrep:

- depending on concurrent accesses mmap can just sigbus on you (e.g. if the mapped file is being truncated)

- mmap simply does not work with virtual filesystems, and will blow up on large files on 32b systems (windows also has further limitations on mmaps)

- depending on workload mmap may not be faster than regular reads and memory buffers, ripgrep will mmap when working on just a few files, but will use normal buffers for large file counts, because when you start reusing buffer you amortise allocation costs which you can't amortise when creating and destroying mappings

- not only that but mmap/munmap are also globally blocking on the process, so in multithreaded processes it's very bad to map/unmap a lot, you can stall your own application

- the semantics of mmap on crash are also somewhat risky, in that the OS will try very hard to sync, but that may not be desirable if the application crashes in the middle of a write

Re:Why MMAP in llama.cpp hides true memory usage by smeenai on Monday April 03, 2023 @03:06PM (#35429581)
https://reviews.llvm.org/D69294 is an interesting case we ran into with LLD (the LLVM linker), where the the memory pressure from mmap'ing a large output file combined with filesystem compression resulted in particularly bad performance.
Re:Why MMAP in llama.cpp hides true memory usage by quotemstr on Monday April 03, 2023 @06:54PM (#35432502)
I wouldn't call any of these things a "bug", in the sense of a behavior that departs from an explicit or implicit specification. Some of them are just advantages and disadvantages of that overall approach.

Also, I'm disappointed that VM languages don't turn SIGBUS accessing a memory mapped region into a VM exception. They could, and it wouldn't even be that hard.

Re:Why MMAP in llama.cpp hides true memory usage by Jasper_ on Tuesday April 04, 2023 @10:25AM (#35440167)
LLAMA maps a single read-only file at startup. The only caveat that applies here is the virtual filesystems one, and it's a fair assessment, so a fallback might be warranted, but mmap seems fine by default.