Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants.
Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly.
Llama.cpp also uses the same inference setting defaults regardless which version of the model you use and some settings are simply defaults it uses for all models.
Then even if you account for all of these, you have to be using Gemma 4 itself correctly, which many people do not.
All of these little changes and inconsistencies hurt some of the model's original capabilities. Even if you go directly to Google's repo and download the full float 16 weights with the template they have there now, you cannot simply assume you're getting the best results.
When I tried to run llama.cpp directly I was getting max 9tk/s on qwen3.5-9B, then I tried LM Studio with the same model and got 77tk/s. I haven't figured out yet how to get MTP working properly in either.
Then create your own reasoning tests to verify that it is working correctly. You can set a specific seed value to make sure the generation is the same every time, that way you can identify any tokens that are different.
Afterwards, try making small incremental changes to the template and validate your tests each time in order to try to adopt the improvements from the newer templates. If the reasoning quality degrades, undo your changes and try again or test alternative solutions.
Would appreciate any kind of pointer to the latest!
That is a lot, what is your laptop hardware?
One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled.
Qwen 3.5+ is much better in that regard with its super efficient context. Even on Macs, speeds take degrade much more slowly.
Yes, this is something I hope they will change. Gemma 4 31B is much slower on pre-Blackwell GPUs as a result, which is a bit of a shame for local model experimentation.
Of course it’s possible the labs just stick with the optimal architecture for large models and GDN is best for smaller models.
Is it multimodal? How do you do transcription with it?
Only up to 30s though, and the larger 26B A4B and 31B models are text and image only.
Even more impressively it doesn't have a separate mmproj at all — it is fully integrated, and the vision encoder doesn't speak words into the LLM, as it were —- it is directly integrated into the model's weights.
I have banged on about this model here enough but I really agree that Gemma 4 12B is a candidate for the most impressive LLM of the year. It is remarkable, and I think because it is a small model that isn't apparently excellent for long-context agentic coding, it has been largely ignored.
It is, actually, quite good at coding jobs. (Though its grasp of nuance is a bit weaker. For example, it doesn't know that closures created inside PHP objects have implicit access to the object as $this, and always seems to need reminding.)
If you instead treat it as a prediction of what consumer on-device AI may very soon be able to do, or even as a possible future into a sort of lower-ratio MoE, or the basis of a modest private offline educational LLM model, it's very interesting indeed.
I've learned a lot from it — the fact that it performs so well at such a small size really does help you assess claims made for much larger models, and it's quick enough on my M1 Max to just muck about with.
I do think the release of these models was somewhat fluffed up, and I don't think it helps that the 31B model uses global attention so it underperforms on the kind of older GPUs that are on a lot of desks; it's no better on those than it is on my M1 Max, where other attention schemes seem to be radically better.
Now that tool-calling is mostly fixed, it's well worth playing with them.
As for coding, for sure there are many important details that a model needs to know in order to produce correctness and the smaller a model is the more it ends up training out. If there's a task you do consistently enough though, often times you can simply provide a pile of essential context so it has good enough reference to not need the extra training data.
Gemma4 screwed up a proxmox install I had. I booted to a SystemRescue install and tried to get gemma4 to fix it. It just could not do it and kept having issues where it dropped a linux command into the local powershell because it did not ssh into systemRescue or killed the ssh connection somehow so the text landed on the wrong system.
I told qwen3.8 to investigate fixing the partition. It said information was lost, but displayed enough info that it was easy to tell it was right. I told it to install fresh proxmox and gave a short rundown on settings and partition sizes I wanted. It made a plan and told me I had to manually installed proxmox by booting the iso. I responded with something like "there are other ways to install promox without human interaction so use one of those". That was it. I woke up to the system having booted to a new proxmox install with my previous ssh keys restored and my existing zfs pool already mounted.
I don't see how any model that is limited to a single context window in a single session would be viable for coding. I want something that can manage the entire project and not just individual files or inline suggestions. I need to be able to feed it all the info I would use to make coding decisions and then have it at least make a working project that it can launch and test successfully. You want it to ask as many questions up front to enable continuous work without stopping for human input.
This isn't an isolated incident, really, I find myself always having these issues with the Gemma series. I'm sure they can do useful things for someone else, but for the things I want to use LLMs for (very small code generation, quick questions, code review) they always seem to disappoint me. I'm sure it's because of the stuff that I do and use, but it's a very consistent red thread with these models for me.
Edit:
The same question for Qwen3.6-35B-A3B produces a pretty concise and correct answer that would be useful to the questioner, without even going to the web. I don't know what Gemma models are trained on, but it's not the stuff that's relevant to me.
It can run 80-100t/s on a laptop, can understand images natively and do bounding boxes, read tiny text, understands audio natively as well and can transcribe or translate anything you say, can do accurate long context retrieval with pretty large context windows, tool calling, excellent reasoning and is very token efficient.
It's only 7GB including the mmproj or 8GB with MTP. The Qwen 3.8 27B model Simon was using is ~18GB with MTP+mmproj, rather than 17GB alone. The point is not really that you compare these models directly, but that Gemma 4 12B QAT was really a special moment in model releases deserving of a similar reaction relative to its size, but was mutilated by Google themselves, Unsloth and Llama.cpp.
The overall appreciation I think we're seeing this year in particular is that people are easily surprised when multiple things are improving simultaneously which produce seemingly exponential changes. It isn't just that models are getting smaller, or that reasoning is getting better, or that speculative decoding is becoming mainstream, or that models can understand audio and images better now, or that they can reliably call tools which expands their capabilities, or that context windows are getting larger, or that accurate retrieval is improved, or that.... and so on. It's all of them narrowing in at once that is starting to make local models incredible and truly useful for far more use cases on the existing hardware people already have.