Gemma 4 12B: A unified, encoder-free multimodal model

Re:Gemma 4 12B: A unified, encoder-free multimodal model by superkuh on Wednesday June 03, 2026 @05:20PM (#48390245)

>consumer-grade card with 12G of VRAM and got 5t/s

That speed for token output indicates to me that it somehow is using hybrid mode and involving cpu+system ram somehow. That ~5tk/s is about the ram bandwidth of DDR4 RAM versus that size model at 4bit. Any consumer GPU with 12 GB like a nvidia rtx 2080 or rtx 3060 should be doing 20+ tk/s with llama.cpp and CUDA backend.

[ Parent | on HN ]

Re:Gemma 4 12B: A unified, encoder-free multimodal model by senko on Wednesday June 03, 2026 @05:42PM (#48390536)

Good catch. I haven't looked deeply into it. This is with Vulkan backend on Linux which I understand should be roughly comparable to CUDA? Gfx is rtx 3060(ti?).

I should play a bit more with llama.cpp options and see what bappened there. Thanks!

[ Parent | on HN ]

Re:Gemma 4 12B: A unified, encoder-free multimodal model by superkuh on Wednesday June 03, 2026 @08:10PM (#48391942)

I've had it happen in the past with llama.cpp on linux that the CPU will present itself as a vulkan device GPU1 with "PHYSICAL_DEVICE_TYPE_CPU" and had a mix-up. Might want to try llama-server --list-devices and then append --device Vulkan0 or whatever.

[ Parent | on HN ]

Re:Gemma 4 12B: A unified, encoder-free multimodal model by pja on Thursday June 04, 2026 @04:30PM (#48404206)

The 8 bit quant runs at 36tps using Vulkan on my AMD rx9070.

[ Parent | on HN ]