News for hackers. Stuff that matters.
Re:Virtual memory overhead for trees by neopallium on Saturday August 04, 2012 @09:27PM (#4340260)
You could try to turn on transparent hugepages:

echo always >/sys/kernel/mm/transparent_hugepage/defrag

echo always >/sys/kernel/mm/transparent_hugepage/enabled

For more details see: http://lwn.net/Articles/423584/

Re:Virtual memory overhead for trees by Tuna-Fish on Saturday August 04, 2012 @10:53PM (#4340436)
The transparent huge page code still only gets you 2M/4M pages, which will still fault most of the time when you handle multi-gigabyte in-memory structures. To avoid ever faulting, you really need to use the 1G pages. And no-one is crazy enough to build support for making them transparent. :)
Re:Virtual memory overhead for trees by neopallium on Sunday August 05, 2012 @10:39AM (#4341530)
epistasis called manual management of huge pages a "royal pain in the ass", which is why I suggested trying transparent huge pages since they are easier to work with (i.e. no code change).

I would be interested in seeing some benchmarks that shows the affect of using huge pages of different sizes with hash tables or other data structures.

Here is a memory bandwidth benchmark that uses huge pages (only uses the default size which is 2Mbyte on my system): http://blogs.utexas.edu/jdm4372/2010/11/11/optimizing-amd-op...

It only sums the values from 32 million doubles (about 256Mbytes of ram). So it is only useful as a memory read speed benchmark. But is does show a lot of different ways to optimize memory bandwidth (prefetch, large pages, sse).

Re:Virtual memory overhead for trees by Andys on Sunday August 05, 2012 @05:19AM (#4341011)
We are not really asking for them to be transparent - TFA is talking about low-level kernel fiddling to achieve what would take a couple of lines for a custom hugepage size?
Re:Virtual memory overhead for trees by Tuna-Fish on Sunday August 05, 2012 @09:39AM (#4341394)
I was specifically replying to neopallium, who suggested turning on transparent hugepages, which, however, would not really do any good at all in this situation. Using (non-transparent) 1G large pages is the correct approach here.
Re:Virtual memory overhead for trees by epistasis on Sunday August 05, 2012 @02:40PM (#4342194)
Thanks, I had not seen that article, but LWN's coverage of huge pages is what made me purchase a subscription, it was absolutely fantastic documentation.

It does turn out that to fully remove TLB misses I'd need 1GB pages, which as was pointed out, are rather scary to imagine being paged to disk, as it would take approximately 10s to page it out. As machines with 512GB or 1TB of RAM become more commonplace, I wonder if there could be a mechanism to better assist with their allocation. Right now, one has to use a bunch of manual management and be extremely aware of any other processes that might start consuming memory.