News for hackers. Stuff that matters.

A comment on Accelerating Gemma 4: faster inference with multi-token prediction drafters. Parent comment

Re:Accelerating Gemma 4: faster inference with multi-token p... by mark_l_watson on Wednesday May 06, 2026 @09:07AM (#48035786)
I have been experimenting with using Claude Code with both the qwen3.6 31B MOE and 28B dense models. Yesterday the 31B model once got confused on refactoring some Prolog code and took a very long time to get it right. Functionality for coding or refactoring Python or TypeScript is usually good. but runs slowly on my 32B MacMini.

Ollama has initial support for bf16 MTP Gemma 4 https://ollama.com/library/gemma4:31b-coding-mtp-bf16 but I have to wait for a smaller model.

I understand why people get excited by having the strongest AI to play/work with but economic factors of inference really count also.

[ Parent | on HN ]
Re:Accelerating Gemma 4: faster inference with multi-token p... by gertlabs on Wednesday May 06, 2026 @02:32PM (#48039842)
If the goal is cost savings, a good compromise is DeepSeek V4 Flash. It'll handle context and agentic work better than any model that can fit on a laptop.
[ Parent | on HN ]