Accelerating Gemma 4: faster inference with multi-token prediction drafters

Re:Accelerating Gemma 4: faster inference with multi-token p... by zmmmmm on Wednesday May 06, 2026 @05:41AM (#48034262)

nice ... i think i get the idea - it's effectively the same / similar benefit as batching, but you're batching against your own speculated future path. Which would be pointless if you didn't have a high probability path to evaluate against - but the draft gives you that.

[ Parent | on HN ]

Re:Accelerating Gemma 4: faster inference with multi-token p... by esyir on Wednesday May 06, 2026 @08:13AM (#48035316)

I'll add an expansion here. It's more useful to you locally, as you have excess compute that's generally wasted. If you're serving multiple user and trying to max output, you might cost some in this case

[ Parent | on HN ]

Re:Accelerating Gemma 4: faster inference with multi-token p... by nullc on Saturday May 09, 2026 @11:14AM (#48075634)

An obvious thing to do is that if you have enough concurrent batches to max out performance you should use those and not speculate. But if compute would be idle waiting on memory, fill the excess with speculation.

[ Parent | on HN ]