Assuming you've also set overcommit_ratio to a conservative value, this would mean that the kernel overcommit=2 setting is not working, since mmap-ing writable pages counts as allocation from overcommit POV. It's a kernel sysctl, so I wouldn't expect it to change glibc's behaviour. There don't seem to be any open bugs about this in the kernel bugzilla. But do tell if you have found a bug in this area.

(This is documented in https://www.kernel.org/doc/html/latest/mm/overcommit-account... )