With overcommit disabled, fork may fail, but your other claims are backwards: malloc can fail, and processes aren't killed.

I suggest trying that :) You'd be surprised.

glibc allocator, like pretty much any other allocator, maps pages in advance without touching them. It's quite likely that you'll get OOM killed when you first try to touch them, rather than during mmap() for new pages.

Assuming you've also set overcommit_ratio to a conservative value, this would mean that the kernel overcommit=2 setting is not working, since mmap-ing writable pages counts as allocation from overcommit POV. It's a kernel sysctl, so I wouldn't expect it to change glibc's behaviour. There don't seem to be any open bugs about this in the kernel bugzilla. But do tell if you have found a bug in this area.

(This is documented in https://www.kernel.org/doc/html/latest/mm/overcommit-account... )

> malloc() in Linux will never fail for small allocations

That's in kernel-space, not user-space processes. If you have overcommit disabled, it will be user-space processes making syscalls mmap() or sbrk() to get more heap memory (in which small allocations reside), that fails with an error return code, I assume. But I guess it could also fail to auto-grow the main thread stack, and presumably get a segfault. If the kernel needs to allocate more memory for a fork(), that syscall can fail. But if the kernel needs to allocate memory for some other internal purpose, probably back to the OOM killer, I guess ... hopefully this would be very rare?

> If you have overcommit disabled, it will be user-space processes making syscalls mmap() or sbrk() to get more heap memory (in which small allocations reside), that fails with an error return code, I assume.

Nope. No overcommit simply means that the OOM killer will be invoked earlier. The errors likely won't happen on the synchronous path due to the way glibc() maps the pages.

Long ago, I tried to write an OOM-resilient container library, and I wanted to test it. It worked as expected on Windows, but on Linux I was not able to get NULLs from malloc(). No amount of MLOCKALL and other tricks helped.

Wow, that's disappointing. It seems there's really a lot of reasons to just accept overcommit on linux. (Browers, js engines used in various components now, databases that fork, many existing libraries not expecting allocations to fail ... and I guess they really don't after all?!)

I think to fix this stuff you'd have to time travel back to 1980 or so. There's just so much software dependent on the broken behavior and a culture of not fixing things.