News for hackers. Stuff that matters.

A comment on Accelerating Gemma 4: faster inference with multi-token prediction drafters. Parent comment

Re:Accelerating Gemma 4: faster inference with multi-token p... by m12k on Wednesday May 06, 2026 @03:18AM (#48033271)
So we've basically taken the concept of branch prediction from CPUs and applied it to LLMs?
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by c7b on Wednesday May 06, 2026 @03:32AM (#48033343)
The concept of predicting future elements in a series is not specific to CS. It's older than computers.
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by fragmede on Wednesday May 06, 2026 @03:23AM (#48033293)
Well, the TPUs they're running on don't have branch prediction, so that had to end up somewhere in the stack.
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by mike_hearn on Wednesday May 06, 2026 @04:57AM (#48033922)
Maybe at very high level of abstraction, but there's no branching involved.
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by lossolo on Wednesday May 06, 2026 @07:46AM (#48035128)
Well, there are multiple token proposals processed in parallel, from which only one is picked, seems like branching to me. The only difference is that in case of CPU there is always only one possible branch that is correct.
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by monster_truck on Wednesday May 06, 2026 @09:20AM (#48035910)
Well, not exactly, but that was the dream we were sold (here be dragons)
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by kpw94 on Wednesday May 06, 2026 @12:44PM (#48038350)
Speculative execution techniques in software & hardware exist everywhere,

- Speculative multi threading

- Data Value Speculation

- Speculative Memory Disambiguation

- Runahead Execution

- Speculative Prefetching

- Multi-path (Dual-path) Execution (goes beyond branch prediction by computing both paths)

- Optimistic Concurrency Control (for database transactions etc)

[ Parent | on HN ]