News for hackers. Stuff that matters.
Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @02:13AM (#43943560)
Two things:

1. Because the support in llama.cpp is horizontal integrated within ggml ecosystem, we can optimize it to run even faster than ollama.

For example, pixtral/mistral small 3.1 model has some 2D-RoPE trick that use less memory than ollama's implementation. Same for flash attention (which will be added very soon), it will allow vision encoder to run faster while using less memory.

2. llama.cpp simply support more models than ollama. For example, ollama does not support either pixtral or smolvlm

Re:Vision Now Available in Llama.cpp by roger_ on Saturday May 10, 2025 @03:26AM (#43943857)
Won’t the changes eventually be added to ollama? I thought it was based on llama.cpp
Re:Vision Now Available in Llama.cpp by diggan on Saturday May 10, 2025 @07:28AM (#43944833)
As far as I understand (not affiliated, just a user who peeked at the code), Ollama started out using llama.cpp as a runner for everything. But eventually they wrote their own runner in Golang, which is where they add support for new models. So most models you run via Ollama uses llama.cpp, but new stuff their own Golang runner.
Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @02:22AM (#43943597)
By the way - fantastic work again on llama.cpp vision support - keep it up!!
Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @02:41AM (#43943660)
Thanks Daniel! Kudos for your great work on quantization, I use the Mistral Small IQ2_M from unsloth during development and it works very well!!
Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @02:54AM (#43943715)
:)) I did have to update the chat template for Mistral - I did see your PR in llama.cpp for it - confusingly the tokenizer_config.json file doesn't have a chat_template, and it's rather in chat_template.jinja - I had to move the chat template into tokenizer_config.json, but I guess now with your fix its fine :)
Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @03:01AM (#43943745)
Ohhh nice to know! I was pretty sure that someone already tried to fix the chat template haha, but because we also allow users to freely create their quants via the GGUF-my-repo space, I have to fix the quants produces from that source
Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @03:23AM (#43943845)
Glad it all works now!
Re:Vision Now Available in Llama.cpp by nolist_policy on Saturday May 10, 2025 @04:39AM (#43944130)
On the other hand ollama supports iSWA for Gemma 3 while llama.cpp doesn't. iSWA reduces kv cache size to 1/6.
Re:Vision Now Available in Llama.cpp by vlovich123 on Saturday May 10, 2025 @05:18AM (#43944300)
What’s iSWA? Can’t find any reference online
Re:Vision Now Available in Llama.cpp by nolist_policy on Saturday May 10, 2025 @06:13AM (#43944500)
interleaved sliding window attention
Re:Vision Now Available in Llama.cpp by imtringued on Saturday May 10, 2025 @06:19AM (#43944530)
Gemma 3 has some layers with a context size of 1024 tokens and others having full length. You need to read the Gemma technical report.