News for hackers. Stuff that matters.
Re:Accelerating Gemma 4: faster inference with multi-token p... by el_isma on Tuesday May 05, 2026 @03:38PM (#48027446)
How is this different from the speculative decoding that we had before?

You could pair a big and small model like qwen 32b with qwen 4b and had that same dynamic of the small model generating tokens and the big one "certifiying" them.

The blog says something about re-using the big model's data?

Re:Accelerating Gemma 4: faster inference with multi-token p... by adrian_b on Tuesday May 05, 2026 @04:11PM (#48027832)
Multi token prediction is the same thing as speculative decoding. This is mentioned in the Google pages describing their MTP implementation.

Google has now provided small models for each of the previous Gemma 4 models, e.g. "gemma-4-26B-A4B-it-assistant" for "gemma-4-26B-A4B-it".

The difference vs. Qwen is that here each small model is not some general-purpose smaller model, but a model that has been optimized specifically for this task, to predict the output of the bigger model with which it is paired.

This specialization and optimization of the Google "gemma-4-*-assistant" models ensures that they are much smaller and thus much faster than general-purpose small models.

Re:Accelerating Gemma 4: faster inference with multi-token p... by julianlam on Wednesday May 06, 2026 @12:29AM (#48032171)
So then these models could be used by llama.cpp today with the -md switch?

Interesting, must try tomorrow.

Re:Accelerating Gemma 4: faster inference with multi-token p... by fulafel on Wednesday May 06, 2026 @08:30AM (#48035455)
Multi-token prediction is a refined form of speculative decoding.

Researchers at Google came up with Speculative decoding in 2022: https://research.google/blog/looking-back-at-speculative-dec... (Fast Inference from Transformers via Speculative Decoding - Yaniv Leviathan, Matan Kalman, Yossi Matias)

Researchers at Meta came up with MTP, a smarter way of doing speculative decoding in 2024: https://arxiv.org/abs/2404.19737 (Better & Faster Large Language Models via Multi-token Prediction Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve)

DeepSeek V3 shipped MTP in a product first, in 2024: https://arxiv.org/abs/2412.19437 (DeepSeek-V3 Technical Report, 100+ authors)

Re:Accelerating Gemma 4: faster inference with multi-token p... by OneDeuxTriSeiGo on Tuesday May 05, 2026 @04:00PM (#48027682)
As far as I can tell MTP is unique from regular speculative decode because the small model is trained to consume and operate on the big model's hidden state for prediction.
Re:Accelerating Gemma 4: faster inference with multi-token p... by dchftcs on Wednesday May 06, 2026 @07:09AM (#48034850)
It's the same speculative decoding. The news is that it came out for a popular local model.