With mmap()-ed file, for each pagefault, kernel will conservatively estimate block size to page in, so you'll have a ton of relatively small requests going to SSD. This would be IOPS-bound, and likely under-perform relative to maximum possible bytes/second throughput.
With explicit read()/pread(), kernel & SSD can work with much larger chunks, so it's easier to hit maximum bytes/second throughput.
Plus, with modern CPUs, IO-wait could be efficiently combined with number-crunching. So, if software knows in advance which data chunk (expert) it'll need for the next token, it can load that in parallel with computing current token.
You could actually use the model's MTP head to make a ~decent prediction on what experts would be activate in future tokens and preload them
Yeah, I was thinking MADV_WILLNEED might work there but not sure
And there is some reuse. ~41% selected again for the next token, ~57% within two. Each layer has its own experts, so no reuse between these layers.
mmap benchmark did basically page touch experiment and cold reads were much slower, unfortunately (10ms vs 3ms)
I tried MADV_WILLNEED, F_RDADVISE and preadv. preadv reduced parallelism because requested experts are rarely adjacent in the file.
pread is still the fastest. And I think Flash-Moe got the same result too
With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work
And common weights still use mmap for simplicity
So, I believe llama.cpp might run it under 2gb, but I assume it will be slower