Re:Fixing a Tough Memory Leak in Python
by glandium on Saturday December 08, 2018 @11:28AM
(#18635962)
Python uses its own allocator for small sizes (below 512 bytes if I remember correctly). I've had a similar problem to yours. See https://glandium.org/blog/?p=3698 and the followup
https://glandium.org/blog/?p=3723 . I have a glibc bug open about the issue https://sourceware.org/bugzilla/show_bug.cgi?id=23416 .
[ Parent
| on HN ]
Re:Fixing a Tough Memory Leak in Python
by MrRadar on Saturday December 08, 2018 @01:22PM
(#18636624)
Wow, I've experienced an issue that's very similar to that one if not the same. Using malloc/free directly worked just fine on Windows and older versions of glibc but lead to runaway RSS on at least some newer versions. I ended up writing a small custom allocator on top of mmap to work around it. It's incredibly frustrating when the memory leak is in the system allocator and not your code.
Re:Fixing a Tough Memory Leak in Python
by X-Istence on Saturday December 08, 2018 @03:18PM
(#18637285)
We at Crunch.io run Python using jemalloc because we have found some really bad behaviour in the default glibc malloc when using Python with many threads.
Using jemalloc we were able to reduce our memory pressure over time dramatically.