News for hackers. Stuff that matters.

A comment on Accelerating Gemma 4: faster inference with multi-token prediction drafters. Parent comment

Re:Accelerating Gemma 4: faster inference with multi-token p... by mike_hearn on Wednesday May 06, 2026 @04:57AM (#48033922)
Maybe at very high level of abstraction, but there's no branching involved.
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by lossolo on Wednesday May 06, 2026 @07:46AM (#48035128)
Well, there are multiple token proposals processed in parallel, from which only one is picked, seems like branching to me. The only difference is that in case of CPU there is always only one possible branch that is correct.
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by monster_truck on Wednesday May 06, 2026 @09:20AM (#48035910)
Well, not exactly, but that was the dream we were sold (here be dragons)
[ Parent | on HN ]