Disable transparent hugepages

Re:Disable transparent hugepages by lorenzhs on Tuesday November 28, 2017 @09:18AM (#15797473)

I've had a really bad run-in with transparent hugepage defragmentation. In a workload consisting of many small-ish reductions, my programme spent over 80% of its total running time in pageblock_pfn_to_page (this was on a 4.4 kernel, https://github.com/torvalds/linux/blob/v4.4/mm/compaction.c#...) and a total of 97% of the total time in hugepage compaction kernel code. Disabling hugepage defrag with echo never > /sys/kernel/mm/transparent_hugepage/defrag lead to an instant 30x performance improvement.

There's been some work to improve performance (e.g. https://github.com/torvalds/linux/commit/7cf91a98e607c2f935d... in 4.6) but I haven't tried if this fixes my workload.

[ Parent | on HN ]

Re:Disable transparent hugepages by mlrtime on Tuesday November 28, 2017 @09:40AM (#15797619)

Did you try allocating hugepages statically at startup? This will also remove the fragmentation.

[ Parent | on HN ]

Re:Disable transparent hugepages by lorenzhs on Tuesday November 28, 2017 @10:11AM (#15797897)

The algorithm was implemented in a big data framework that handles the allocations, so I would have needed to significantly adapt its memory subsystem to change this. I've talked to the authors, though, and it's not easy to change. Easier to disable transparent hugepage defrag, especially when there's a paper deadline to meet :)

[ Parent | on HN ]