News for hackers. Stuff that matters.
Re:Fixing a Tough Memory Leak in Python by glandium on Saturday December 08, 2018 @11:28AM (#18635962)
Python uses its own allocator for small sizes (below 512 bytes if I remember correctly). I've had a similar problem to yours. See https://glandium.org/blog/?p=3698 and the followup https://glandium.org/blog/?p=3723 . I have a glibc bug open about the issue https://sourceware.org/bugzilla/show_bug.cgi?id=23416 .
Re:Fixing a Tough Memory Leak in Python by MrRadar on Saturday December 08, 2018 @01:22PM (#18636624)
Wow, I've experienced an issue that's very similar to that one if not the same. Using malloc/free directly worked just fine on Windows and older versions of glibc but lead to runaway RSS on at least some newer versions. I ended up writing a small custom allocator on top of mmap to work around it. It's incredibly frustrating when the memory leak is in the system allocator and not your code.
Re:Fixing a Tough Memory Leak in Python by X-Istence on Saturday December 08, 2018 @03:18PM (#18637285)
We at Crunch.io run Python using jemalloc because we have found some really bad behaviour in the default glibc malloc when using Python with many threads.

Using jemalloc we were able to reduce our memory pressure over time dramatically.