Wow, I've experienced an issue that's very similar to that one if not the same. Using malloc/free directly worked just fine on Windows and older versions of glibc but lead to runaway RSS on at least some newer versions. I ended up writing a small custom allocator on top of mmap to work around it. It's incredibly frustrating when the memory leak is in the system allocator and not your code.