Well... because it's not a myth in all cases?
$ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap
real 1.167
user 0.815
sys 0.349
maxmem 9473 MB
faults 0
$ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap
real 1.748
user 0.506
sys 1.239
maxmem 9 MB
faults 0
> ripgrep may abort unexpectedly when using
> default settings if it searches a file that
> is simultaneously truncated. This behavior
> can be avoided by passing the --no-mmap flag
> which will forcefully disable the use of
> memory maps in all cases.
Changing the workload can dramatically alter these conclusions. For example, on a checkout of the Linux kernel:
$ time rg zqzqzqzq --mmap
real 1.661
user 1.603
sys 3.128
maxmem 41 MB
faults 0
$ time rg zqzqzqzq --no-mmap
real 0.126
user 0.702
sys 0.586
maxmem 20 MB
faults 0
FWIW, I do generally disagree with your broader point, but it's important to understand that there's actually good reason to believe that using mmaps can be faster in some circumstances.
That said, the case isn't as obvious as you make it: you apparently save on copies and explicit system calls, but mmap replaces those with page table manipulation and "hidden" system calls (i.e., page faults).
These page faults have as much per-call overhead as regular system calls, and so if mmap actually faulted in every page, I'm pretty sure it would actually be slower than read(), since read with a 16K buffer (for example) would make only 25% as many syscalls as mmap bringing in every 4K page.
On modern Linux, by default, mmap doesn't fault in every page, due to "faultaround" which tries to map in additional nearby pages every time it faults (16 by default), so the number of faults is 1/16th what you'd expect if it faulted on every page. You can avoid additional mapping on access with MAP_POPULATE or madvise (? maybe) calls, but then this introduces the same kind of window management problem as read: you lose the abstraction of the entire file just mapped into memory.
Beyond that, mmap has to do "per page" work to map the file: adjusting VM and OS structures to map the page into the process address space, and then undoing that work on munmap (which is the more expensive half since it includes TLB invalidation, and possibly a cross-core shootdown). You'd thing that this work would be much faster than copying 4 KiB of memory, but it isn't - and on some systems with small pages sizes and/or slow TLB invalidations it can be slower overall.
> It should be faster in nearly all circumstances
As my previous comment showed, that's definitely not true. If you're searching a whole bunch of small files in a short period of time, then it appears that the overhead of memory mapping leads to a significant performance regression when compared to standard `read` calls.
> it’s really the easy case
I know. :-) That's why ripgrep has both modes. It chooses between them based on the predicted workload. It uses memory maps automatically if it's searching a file or two, but otherwise falls back to standard read calls.
Moreover, if ripgrep aborts once in a while because of a SIGBUS, then it's usually not a big deal. It's fairly rare for it to happen. And if it does happen to you a lot or you never want it to happen, then you just need to `alias rg="rg --no-mmap"`.
I was pondering this some more in the shower, the mmap for rg case is also sort of naturally cache oblivious, copies will consume hardware cache for the write and while there is a ton of hardware for cache on modern hardware, it’s a noticeable cost on some tests. If you’re searching through something big, then it’d be like doubling hardware cache which is probably really noticeable on smaller devices.
The small files case is interesting, copying the data is faster than patching up the page table tree, I bet there is a strong correlation to the hardware cache size vs the average file size in that case. The files probably need to be N pages in size for it to be worth it, might be an interesting heuristic to use.
But for general production case mmap shouldn't even be considered a solution to the syscall and memory copy overhead problem. If that overhead is too big for you, other approaches work better, like buffering, application level caching, etc.
Buffering and application level caching mean you're wasting memory, and also wasting code space and CPU time because you're duplicating work that the OS already does.
The first numbers seem to imply that it takes equally long for pread to copy bytes from memory as it does to fetch them from the disk. For a quick back-of-the-napkin attempt at checking this, lets assume that disk IO accounts for 100% of this workload, and that local memory is one order of magnitude faster. In that case, I would expect the difference for an optimized implementation to be at most 10%.
I do think it is true that there are scenarios where the file mmap is faster, or that certain operations on each kernel might fall off a cliff. I just find it hard to believe that `mmap` must be as much faster as shown here in a typical situation (e.g. after a clean reboot, doing about the same amount of work, issuing optimal syscalls, with the OS/kernel not doing anything foolish).
That is indeed a very common case for ripgrep, where you might execute many searches against the same corpus repeatedly. Optimizing that use case is important.
For cases where the file isn't cached, then it's much less interesting, because you're going to just be bottlenecked on disk I/O for the most part.
> then we might be just comparing shared memory against IPC, and that's an obvious performance win, but not really what's intended to be examined here.
Please don't take my comment out of context. I was specifically responding to this fairly broad sweeping claim with actual data:
> This old myth that mmap is the fast and efficient way to do IO just won't die.
You might think the fact that this isn't a myth is "obvious," but clearly, not everyone does. The right remedy to that is data, not back of the napkin theorizing. :-)
If you want to try your own benchmarks in your own environment, then you can: https://github.com/BurntSushi/ripgrep/
On Linux at least, you do not need to do a clean reboot to measure something without cache. You can drop the file cache with `sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`.
Can you link to the libc maintainer responses? I would like to know how they countered you exactly.
In my experience, it's pretty easy to write async-signal-safe code --- which is almost always also thread-safe --- if you follow a few simple rules. I don't understand why people have so much difficulty writing correct signal handlers.
Thanks for interesting read. That said, I can see why glibc people didn't appreciate the proposal.
The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. There are some dangerous-sounding paragraphs too:
> It’s occasionally useful to longjmp out of a signal handler. It’s reasonable to want to return non-locally from a shared signal handler too --- that is, to resume program execution after SIGNAL_CONTINUE_EXECUTION in a different state from the state the program had when we entered the shared signal handler. Since the signal system probably wants to maintain some kind of state to track its progress through its shared signal handler list, a plain longjmp out of a shared signal handler will likely leave the system in an unspecified state.
Are you sure, that we should worry about "signal system maintaining some kind of state"? Not about rest of application being in completely unspecified state?!!
The article proposes a primitive system for setting signal priorities, but stops at a half-baked solution. There is a mention of banning longjmp, but individual handlers still can bail via SIGNAL_CONTINUE_EXECUTION. What if I want my handler to always run regardless of registration order?
The proposal does not offer a way to retrieve a list of already installed handlers, which makes that part of it even worse than existing Posix signal API.
The proposed API does not address challenges of using signals in multi-threading programs.
The article mentions, that signal handlers can't be reliably unloaded, but proposed API does not address it.
Overall the proposed interface brings little to the table, does not work well alongside with existing sigaction() API and creates false illusion, that signal handlers are safe and ok to use. I imagine, that if it had more technical "meat" — more like robust mutexes or FD_CLOEXEC — it would have seen a lot more constructive discussion and less hostility from glibc maintainers.
If you're writing a signal handler, the signal handler needs to be async-signal-safe. You can't just wave your arms and make the problem of async signal safety disappear, because CPU traps themselves are async-signal-unsafe. Even userfaultfd has to deal with async signal safety issues, since a thread causing a fault can, in principle, be anywhere!
> Are you sure, that we should worry about "signal system maintaining some kind of state"? Not about rest of application being in completely unspecified state?!!
If your handlers play by the rules and are async-signal-safe, there's no problem.
> The article proposes a primitive system for setting signal priorities, but stops at a half-baked solution.
What's half-baked about it?
> What if I want my handler to always run regardless of registration order?
That's a logical nonsense request. What if two components want to their handlers to be the highest priority?
> The proposal does not offer a way to retrieve a list of already installed handlers, which makes that part of it even worse than existing Posix signal API.
The whole point of the facility is to let different components share a signal without stepping on each other. Why would you need to retrieve the list of handlers?
> The proposed API does not address challenges of using signals in multi-threading programs.
This claim is too vague to rebut. What specific "challenges" are you talking about?
> The article mentions, that signal handlers can't be reliably unloaded, but proposed API does not address it.
The proposed API works fine with library unloading: a library can unregister whatever handlers it's installed just before being unloaded (e.g., in a static destructor), and this unregister operation is guaranteed to be safe no matter what the order handlers are unloaded.
> Overall the proposed interface brings little to the table
It allows multiple components to safely share signals. The objections you've mentioned are based on your misunderstanding my proposal.
> does not work well alongside with existing sigaction()
Yes it does. The article talks about this interaction specifically.
> I imagine, that if it had more technical "meat"
The glibc people literally think that nobody should be using signals. That's their objection, not anything you've talked about.
I don't believe, that everyone holds that opinion. Even if they did, the world does not revolve around Red Hat's team, — there are still kernel mail lists and other venues for discussion. But if proposed improvements aren't well thought-out, would anyone there back them up?
In my opinion, async-signal safety in itself is much bigger problem than robust registration of signals. The later is mostly solved by chaining signal handlers, while former is mostly unsolved (and keeps getting worse). Proliferation of new libraries and async-signal unsafe conventions. People keep using printf() in signal handlers. Occurrences of fork() in multi-threaded apps. Still no async-signal safe malloc() (some Googlers tried, but the idea didn't get much traction). And then you come and propose new interface for registering signals handlers, and say that "It’s okay for two functions can be async-signal-unsafe". If your proposed API is async-signal unsafe, how would it deal with signals arriving during dispatch of signal handler list?
I think the proposal is thought-out. It's the objections I've seen that suggest a lack of thorough consideration.
> the world does not revolve around Red Hat's team
The facility I'm describing needs to be in libc to be useful, and for better or worse, if it's not in glibc and isn't something you can use as a raw system call, it might as well not exist.
> In my opinion, async-signal safety in itself is much bigger problem than robust registration of signals
Async signal safety concerns are inherent in any approach that exposes CPU traps to userspace, since traps occur at instruction granularity. As I've said elsewhere, writing async-signal-safe code isn't that hard if you follow a few basic rules. You can't solve the async signal safety problem, and we shouldn't need an all-singing, all-dancing async-signal-safety-requirement-avoidance system just to improve on the signal API.
The alternative to what I'm proposing isn't that everyone abandons signals. The alternative is that everyone keeps using sigaction, which sucks.
> The later is mostly solved by chaining signal handlers,
No, it isn't. I go into great detail in my note, in which I explain why current solutions to this problem are lacking and describe ways we can do better.
> Proliferation of new libraries and async-signal unsafe conventions. People keep using printf() in signal handlers. Occurrences of fork() in multi-threaded apps.
Okay, so don't do those things. The problems you're describing all come from people ignorant of safe signal-handler programming practices writing bad code. The problems disappear with education. The problems I'm describing don't disappear with education, since the current signal API imposes unavoidable limitations that even the best code can't work around, even in principle.
It's as if we have a car with hand-cranked windows and no brakes and you're annoyed that we would fix the brakes before adding power windows. The brakes are necessary to drive the car properly. Power windows are just an ergonomic feature.
> If your proposed API is async-signal unsafe, how would it deal with signals arriving during dispatch of signal handler list?
I don't understand this objection. Registration doesn't have to be async-signal-safe. Dispatch must be. There are multiple ways of implementing such a system --- e.g., CAS on a word pointing to a signal control structure.
(Yes, you can reduce the severity of this problem with MADV_DONTNEED and friends.)
That said, there's nothing wrong with mmap or SIGBUS error reporting in concept. I think I've said it before, but you can think of mmap of a disk file as basically adding a temporary dedicated swap file to the system, with all the pages backed by that "swap file" already "swapped" out unless already cached. Sometimes that's exactly what you want!
The author's signal handler registration difficulties come from the POSIX signal API being awful, not from the idea that catching CPU traps is somehow bad. It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1].
When I ran [1] by the glibc people, though, the response from them was pretty stark and unwarranted hostility toward any use of signals at all, even in perfectly legitimate scenarios, like mmap SIGBUS or certain kinds of important JVM-style pointer check [2] optimizations. It's this "we won't change anything anywhere despite our views being out of step with real user needs" attitude that's responsible for a lot of weird friction in the Unix API surface.
Practically every program mmaps disk files has trouble recovering from IO errors. If it were easier for multiple components in a process to share responsibility for handling a signal, we'd more often get subtleties like this right. A better signal API will improve the user experience. Tilting at signal-shaped windmills won't.
[1] https://www.facebook.com/notes/daniel-colascione/toward-shar...
[2] Relying on CPU traps lets you perform certain checks for free. Why write "if (myptr != nullptr) var = myptr" when you can just write "var = myptr" and handle the SIGSEGV when myptr ends up being nullptr? The latter option is zero-overhead in the non-nullptr case. It's also much more complicated, but if you're a VM, and you generate millions and millions of these "if (myptr != nullptr)" checks in JITed code, the size and speed win of eliminating the check starts to justify the complexity.