News for hackers. Stuff that matters.

A comment on TCMalloc and MySQL.

Re:TCMalloc and MySQL by jeffdavis on Friday February 22, 2013 @01:23AM (#5263029)
Warning: tcmalloc does not release memory back to the OS, ever:

"TCMalloc currently does not return any memory to the system."[1]

That means if you have many long-running processes, then each of them will consume the maximum amount of memory that it ever has. Not good for a multi-tenant setup.

If it's a dedicated server running one multi-threaded application, maybe that's OK, although I'd be a little bit wary anyway.

I should note that, even if the application doesn't let the memory go, the OS could page out the inactive regions. Not really something that I would like to rely on, though. There are some other caveats also, like it would make memory accounting a little trickier ("Wow, that process is huge! Oh, never mind, it's mostly paged out.").

For what it's worth, I just spent considerable effort to get rid of tcmalloc due (in part) to problems like this. [2]

[1] http://goog-perftools.sourceforge.net/doc/tcmalloc.html

[2] You wouldn't think it would be a lot of effort, but we were using dynamic libraries that were linking against tcmalloc, which is outright dangerous if the main executable isn't linked against tcmalloc (you don't want to replace the allocator in a running executable). And some of those libraries were actually using the tcmalloc-specific features/symbols, so I had to get away from that first.

[ Parent | on HN ]
Re:TCMalloc and MySQL by sghemawat on Friday February 22, 2013 @02:22AM (#5263159)
About [1] Sorry about that: the document you linked to is amazingly stale. tcmalloc has been releasing memory to the system for many years. See for example the IncrementalScavenge routine in a version of page_heap.cc from Dec 2008:

https://code.google.com/p/gperftools/source/browse/trunk/src...

One caveat: physical memory and swap space is released, but the process's virtual size will not decrease since tcmalloc uses madvise(MNONE) to release memory.

About [2], code using tcmalloc-specific features/symbols is definitely a problem. I would strongly advise against doing that and sticking to the libc interfaces instead for the reason you pointed out.

[ Parent | on HN ]
Re:TCMalloc and MySQL by jeffdavis on Friday February 22, 2013 @02:28AM (#5263173)
Strange, the page showed up first when I googled "tcmalloc" and the problem was also present in the version that I was using (at least I think it was). My apologies.

Yeah, regarding [2], that was definitely not my idea.

[ Parent | on HN ]
Re:TCMalloc and MySQL by sghemawat on Friday February 22, 2013 @02:53AM (#5263221)
Not your fault. We just plain forgot to update the documentation, so the freshest available document is a few years out of date.
[ Parent | on HN ]
Re:TCMalloc and MySQL by jeffffff on Friday February 22, 2013 @02:42AM (#5263198)
wow i didn't realize madvise could actually modify the memory contents and return pages, but it makes sense that it can because that is a useful feature. very cool!
[ Parent | on HN ]
Re:TCMalloc and MySQL by jeffffff on Friday February 22, 2013 @02:18AM (#5263150)
i don't know of any malloc implementations that return memory to the system. the only way to do that is to pass a negative value to sbrk, which requires all the memory being returned to be at the end of the data segment. even if you free 99% of the memory you were using, if one byte is still in use at the end of the data segment no memory can be returned. this is almost always the case in practice, so no malloc implementations bother to return memory in this way. on the other hand, unmapping an anonymous mmap does return the memory to the system. most mallocs handle large allocations by delegating to anonymous mmap for this reason. you probably could've tweaked the mmap threshold for tcmalloc to get the same result.

edit: apparently tcmalloc is using mmap to allocate more of its memory than i realized, not sure why i thought it was using sbrk for everything

[ Parent | on HN ]
Re:TCMalloc and MySQL by priteau on Friday February 22, 2013 @05:09AM (#5263591)
OpenBSD's malloc implementation does: "On a call to free, memory is released and unmapped from the process address space using munmap."

http://en.wikipedia.org/wiki/C_dynamic_memory_allocation#Ope...

[ Parent | on HN ]
Re:TCMalloc and MySQL by gsg on Friday February 22, 2013 @02:20AM (#5263152)
You mean address space. The OS can and will reclaim all the memory it likes, when it likes.

Pages that are mapped but are left untouched for a long time aren't problematic in modern systems. There is a small cost for the PTE but nothting like an entire page of physical memory.

[ Parent | on HN ]
Re:TCMalloc and MySQL by jeffdavis on Friday February 22, 2013 @02:33AM (#5263179)
I was talking about memory that was used in the past, but is no longer used.

The OS can either keep it resident or swap it out, but it has no way of knowing that it is no longer in use (short of something like madvise()). In the realm of sanity, the OS can't just arbitrarily throw away memory that a process has written to (unless it also kills the process).

[ Parent | on HN ]