News for hackers. Stuff that matters.
Re:Virtual memory overhead for trees by Tuna-Fish on Saturday August 04, 2012 @10:53PM (#4340436)
The transparent huge page code still only gets you 2M/4M pages, which will still fault most of the time when you handle multi-gigabyte in-memory structures. To avoid ever faulting, you really need to use the 1G pages. And no-one is crazy enough to build support for making them transparent. :)
Re:Virtual memory overhead for trees by neopallium on Sunday August 05, 2012 @10:39AM (#4341530)
epistasis called manual management of huge pages a "royal pain in the ass", which is why I suggested trying transparent huge pages since they are easier to work with (i.e. no code change).

I would be interested in seeing some benchmarks that shows the affect of using huge pages of different sizes with hash tables or other data structures.

Here is a memory bandwidth benchmark that uses huge pages (only uses the default size which is 2Mbyte on my system): http://blogs.utexas.edu/jdm4372/2010/11/11/optimizing-amd-op...

It only sums the values from 32 million doubles (about 256Mbytes of ram). So it is only useful as a memory read speed benchmark. But is does show a lot of different ways to optimize memory bandwidth (prefetch, large pages, sse).

Re:Virtual memory overhead for trees by Andys on Sunday August 05, 2012 @05:19AM (#4341011)
We are not really asking for them to be transparent - TFA is talking about low-level kernel fiddling to achieve what would take a couple of lines for a custom hugepage size?
Re:Virtual memory overhead for trees by Tuna-Fish on Sunday August 05, 2012 @09:39AM (#4341394)
I was specifically replying to neopallium, who suggested turning on transparent hugepages, which, however, would not really do any good at all in this situation. Using (non-transparent) 1G large pages is the correct approach here.