News for hackers. Stuff that matters.
Re:Memory is slow, Disk is fast – Part 2 by bawolff on Thursday September 04, 2025 @08:26PM (#45133765)
Shouldn't you also compare to mmap with huge page option? My understanding is its presicely meant for this circumstance. I don't think its a fair comparison without it.

Respectfully, the title feels a little clickbaity to me. Both methods are still ultimately reading out of memory, they are just using different i/o methods.

Re:Memory is slow, Disk is fast – Part 2 by jared_hulbert on Thursday September 04, 2025 @09:06PM (#45134007)
The original blog post title is intentionally clickbaity. You know, to bait people into clicking. Also I do want to challenge people to really think here.

Seeing if the cached file data can be accessed quickly is the point of the experiment. I can't get mmap() to open a file with huge pages.

void* buffer = mmap(NULL, size_bytes, PROT_READ, (MAP_HUGETLB | MAP_HUGE_1GB), fd, 0); doesn't work.

You can can see my code here https://github.com/bitflux-ai/blog_notes. Any ideas?

Re:Memory is slow, Disk is fast – Part 2 by jandrewrogers on Thursday September 04, 2025 @10:23PM (#45134410)
Read the man pages, there are restrictions on using the huge page option with mmap() that mean it won’t do what you might intuit it will in many cases. Getting reliable huge page mappings is a bit fussy on Linux. It is easier to control in a direct I/O context.
Re:Memory is slow, Disk is fast – Part 2 by mastax on Thursday September 04, 2025 @09:48PM (#45134269)
MAP_HUGETLB can't be used for mmaping files on disk, it can only be used with MAP_ANONYMOUS, with a memfd, or with a file on a hugetlbfs pseudo-filesystem (which is also in memory).
Re:Memory is slow, Disk is fast – Part 2 by inetknght on Thursday September 04, 2025 @10:28PM (#45134451)
> MAP_HUGETLB can't be used for mmaping files on disk

False. I've successfully used it to memory-map networked files.

Re:Memory is slow, Disk is fast – Part 2 by minitech on Thursday September 04, 2025 @11:07PM (#45134638)
That doesn’t sound like the intended meaning of “on disk”.
Re:Memory is slow, Disk is fast – Part 2 by inetknght on Thursday September 04, 2025 @11:11PM (#45134653)
Kernel doesn't really care about "on disk", it cares about "on filesystem".

The "on disk" distinction is a simplification.

Re:Memory is slow, Disk is fast – Part 2 by pclmulqdq on Thursday September 04, 2025 @11:54PM (#45134845)
The kernel absolutely does care about the "on disk" distinction because it determines what driver to use.
Re:Memory is slow, Disk is fast – Part 2 by ddtaylor on Friday September 05, 2025 @01:23AM (#45135217)
The interface is handled by the kernel.
Re:Memory is slow, Disk is fast – Part 2 by loloquwowndueo on Thursday September 04, 2025 @10:59PM (#45134599)
Share your code?
Re:Memory is slow, Disk is fast – Part 2 by inetknght on Thursday September 04, 2025 @11:07PM (#45134637)
I don't work there any more (it was a decade ago) and I'm pretty busy right now with a new job coming up (offered today).

Do you have kernel documentation that says that hugetlb doesn't work for files? I don't see that stated anywhere.

Re:Memory is slow, Disk is fast – Part 2 by Sesse__ on Friday September 05, 2025 @05:12AM (#45136551)
It's filesystem-dependent. In particular, tmpfs will work. To the best of my knowledge, no “normal” filesystems (e.g., ext4, xfs) will.
Re:Memory is slow, Disk is fast – Part 2 by inetknght on Friday September 05, 2025 @09:57PM (#45145898)
It works fine on my ext4 fs...
Re:Memory is slow, Disk is fast – Part 2 by inetknght on Friday September 05, 2025 @01:06PM (#45140890)
My bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`.

See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb

Re:Memory is slow, Disk is fast – Part 2 by squirrellous on Friday September 05, 2025 @02:38AM (#45135603)
This is quite interesting since I, too, was under the impression that mmap cannot be used on disk-backed files with huge pages. I tried and failed to find any official kernel documentation around this, but I clearly remember trying to do this at work (on a regular ECS machine with Ubuntu) and getting errors.

Based on this SO discussion [1], it is possibly a limitation with popular filesystems like ext4?

If anyone knows more about this, I'd love to know what exactly are the requirements for using hugepages this way.

[1] https://stackoverflow.com/questions/44060678/huge-pages-for-...

Re:Memory is slow, Disk is fast – Part 2 by bawolff on Friday September 05, 2025 @06:06AM (#45136878)
Trying to google this i found https://lwn.net/Articles/718102/ which suggests that there was discussion about it back in 2017. But i can't find anything else about it except a patchset that i guess wasnt merged (?). So maybe it was just a proposal that never made it in.

Honestly i never knew any of this i thought huge pages just worked for all of mmap.

Re:Memory is slow, Disk is fast – Part 2 by inetknght on Friday September 05, 2025 @01:05PM (#45140880)
My bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`.

See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb

Re:Memory is slow, Disk is fast – Part 2 by squirrellous on Saturday September 06, 2025 @03:58AM (#45147460)
Cool! Thanks for the example. The aforementioned work thing requires MAP_SHARED as well which IIRC is the reason it would fail when used together with files and huge pages, but private mappings work as you show.
Re:Memory is slow, Disk is fast – Part 2 by inetknght on Friday September 05, 2025 @01:05PM (#45140875)
My bad, don't use `MAP_HUGETLB`, just use `MAP_HUGE_1GB`.

See a quick example I whipped up here: https://github.com/inetknght/mmap-hugetlb

Re:Memory is slow, Disk is fast – Part 2 by bawolff on Friday September 05, 2025 @04:01PM (#45142964)
The mmap man page kind of implies that would be a no-op, but i haven't tested myself.
Re:Memory is slow, Disk is fast – Part 2 by jared_hulbert on Friday September 05, 2025 @02:05PM (#45141621)
Adding MAP_HUGE_1GB and not MAP_HUGETLB does compile and run for me. Not convinced that its' actually doing anything. Performance is the same.
Re:Memory is slow, Disk is fast – Part 2 by inetknght on Friday September 05, 2025 @02:18PM (#45141792)
Well now that it works, feel free to start poking around at it for a follow-up blog post :)
Re:Memory is slow, Disk is fast – Part 2 by mananaysiempre on Friday September 05, 2025 @02:38AM (#45135606)
It looks like there is in theory support for that[1]? But the patches for ext4[2] did not go through.

[1] https://lwn.net/Articles/686690/

[2] https://lwn.net/Articles/718102/

Re:Memory is slow, Disk is fast – Part 2 by mrlongroots on Friday September 05, 2025 @10:09AM (#45138806)
You don't need hugepages for basic 5GB/s sequential scans. I don't know the exact circumstances that would cause TLB pressure, but this is not it.

You can maybe reduce the number of page faults, but you can do that by walking the mapped address space once before the actual benchmark too.