> If you have overcommit disabled, it will be user-space processes making syscalls mmap() or sbrk() to get more heap memory (in which small allocations reside), that fails with an error return code, I assume.
Nope. No overcommit simply means that the OOM killer will be invoked earlier. The errors likely won't happen on the synchronous path due to the way glibc() maps the pages.
Long ago, I tried to write an OOM-resilient container library, and I wanted to test it. It worked as expected on Windows, but on Linux I was not able to get NULLs from malloc(). No amount of MLOCKALL and other tricks helped.
Wow, that's disappointing. It seems there's really a lot of reasons to just accept overcommit on linux. (Browers, js engines used in various components now, databases that fork, many existing libraries not expecting allocations to fail ... and I guess they really don't after all?!)
That's in kernel-space, not user-space processes. If you have overcommit disabled, it will be user-space processes making syscalls mmap() or sbrk() to get more heap memory (in which small allocations reside), that fails with an error return code, I assume. But I guess it could also fail to auto-grow the main thread stack, and presumably get a segfault. If the kernel needs to allocate more memory for a fork(), that syscall can fail. But if the kernel needs to allocate memory for some other internal purpose, probably back to the OOM killer, I guess ... hopefully this would be very rare?