I use https://lib.rs/cap which self-imposes memory limit on the process. Setting that limit below cgroup limit allows programs to actually handle OOM before they get killed by the OS (if only Rust's libstd wasn't so eager to self-abort anyway).
How do you choose that limit?
For desktop applications it's tough. It may be just an arbitrarily high amount you don't expect to hit during normal operation. If you need to work with variable-size data, then it could be `size_of_file_being_opened * x` if you can predict the `x`.
I wouldn't be surprised if the popular allocators - jemalloc/tcmalloc/scudo or even the defaults in glibc/musl support a setting like this.
Maybe what I'm asking for makes no sense though, because even if my process handles out of memory errors gracefully, it might still get OOM-killed when another process allocates some more.
But at the same time it's critical for kernel and embedded (µC, the "too small for Linux" kind) code.
Isn't it only Linux that does overcommit by default?
The entire problem is, in a way, unsolveable with current OS APIs. AFAIK there is no preexisting, good, actually usable, and universal memory usage meter. Some things work for a lot of cases, but I don't think there's anything that would work universally.
Coincidentally, in my eyes the best way to handle memory pressure in applications would be proactively rather than reactively. Sometimes you can unload more things, like pages in a document that aren't being edited right now. And if you need to fail, you can fail on good boundaries (e.g. refusing entire requests in server-like things) rather than in the middle of something that you need tons of work to unwind correctly.
(Slightly related: pool memory allocators, e.g. APR's https://apr.apache.org/docs/apr/trunk/group__apr__pools.html )
Is there a way to disable it for a single program or cgroup, to enable it to deal with out-of-memory conditions? Maybe changing/hooking the standard library?
mlockall(2) seems overkill, since it will also force all code and mapped files to be resident.