Re:When the memory allocator works against you
by malisper on Sunday March 12, 2017 @05:29PM
(#13853234)
I don't fully understand the post, but I don't think the author found a bug anywhere. The main discovery the author made was that CPython uses its own allocator on top of memory allocated by mmap. This leads to a difference between the reported allocated memory by malloc and the memory used by the program. The author still hasn't discovered a root cause for the large amounts of memory allocated.
[ Parent
| on HN ]
Re:When the memory allocator works against you
by glandium on Sunday March 12, 2017 @06:42PM
(#13853612)
Author here. The last graph actually shows that the problem still happens with CPython's allocator disabled.
Also, to clarify, the amount of memory allocated is actually decreasing (in-use bytes), while the memory allocator requests more and more memory from the system (system bytes), presumably because of fragmentation.
Re:When the memory allocator works against you
by Kenji on Monday March 13, 2017 @03:16PM
(#13861326)
>The last graph actually shows that the problem still happens with CPython's allocator disabled.
Excuse my ignorance, but couldn't it still be a problem of the python runtime? I.e. they use malloc wrong, etc?
Re:When the memory allocator works against you
by malisper on Sunday March 12, 2017 @07:12PM
(#13853843)
I see. I thought system bytes included the mmaped block of memory (the docs for malloc_stats explicitly say it does not) and I thought the last graph was a demonstration of why the Python memory allocator was necessary.