News for hackers. Stuff that matters.

A comment on Accelerating Gemma 4: faster inference with multi-token prediction drafters. Parent comment

Re:Accelerating Gemma 4: faster inference with multi-token p... by cma on Wednesday May 06, 2026 @03:25AM (#48033309)
I think it also gets use in the /fast modes the providers sell at higher cost.
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by gunalx on Wednesday May 06, 2026 @05:37AM (#48034236)
They probably use it on all models. Fast is probably just a resource pool with less congestion and therefore faster throughput per user but less efficent.
[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by cma on Wednesday May 06, 2026 @09:12AM (#48035847)
If it speeds prefill too I guess so.
[ Parent | on HN ]