Re:Show HN: Open-source engine running Gemma 4 26B in 2 GB R...
by p0u4a on Thursday July 30, 2026 @01:59AM
(#49106514)
> if software knows in advance which data chunk (expert) it'll need for the next token, it can load that in parallel with computing current token
You could actually use the model's MTP head to make a ~decent prediction on what experts would be activate in future tokens and preload them
Re:Show HN: Open-source engine running Gemma 4 26B in 2 GB R...
by nijave on Thursday July 30, 2026 @07:19AM
(#49108460)
>So, if software knows in advance which data chunk (expert) it'll need for the next token, it can load that in parallel with computing current token.
Yeah, I was thinking MADV_WILLNEED might work there but not sure
With mmap()-ed file, for each pagefault, kernel will conservatively estimate block size to page in, so you'll have a ton of relatively small requests going to SSD. This would be IOPS-bound, and likely under-perform relative to maximum possible bytes/second throughput.
With explicit read()/pread(), kernel & SSD can work with much larger chunks, so it's easier to hit maximum bytes/second throughput.
Plus, with modern CPUs, IO-wait could be efficiently combined with number-crunching. So, if software knows in advance which data chunk (expert) it'll need for the next token, it can load that in parallel with computing current token.