News for hackers. Stuff that matters.
Re:Vision Now Available in Llama.cpp by simonw on Saturday May 10, 2025 @02:31AM (#43943624)
llama.cpp offers compiled releases for multiple platforms. This release has the new vision features: https://github.com/ggml-org/llama.cpp/releases/tag/b5332

On macOS I downloaded the llama-b5332-bin-macos-arm64.zip file and then had to run this to get it to work:

  unzip llama-b5332-bin-macos-arm64.zip
  cd build/bin
  sudo xattr -rd com.apple.quarantine llama-server llama-mtmd-cli *.dylib
Then I could run the interactive terminal (with a 3.2GB model download) like this (borrowing from https://news.ycombinator.com/item?id=43943370R)

  ./llama-mtmd-cli -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL -ngl 99
Or start the localhost 8080 web server (with a UI and API) like this:

  ./llama-server -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL -ngl 99
I wrote up some more detailed notes here: https://simonwillison.net/2025/May/10/llama-cpp-vision/
Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @02:38AM (#43943650)
For brew users, you can specify --HEAD when installing the package. This way, brew will automatically build the latest master branch.

Btw, the brew version will be updated in the next few hours, so after that you will be able to simply "brew upgrade llama.cpp" and you will be good to go!

Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @02:43AM (#43943667)
I'm also extremely pleased with convert_hf_to_gguf.py --mmproj - it makes quant making much simpler for any vision model!

Llama-server allowing vision support is definitely super cool - was waiting for it for a while!

Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @02:42AM (#43943662)
And btw, -ngl is automatically set to max value now, you don't need to -ngl 99 anymore!

Edit: sorry this is only true on Metal. For CUDA or other GPU backends, you still need to manually specify -ngl

Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @02:44AM (#43943677)
OH WHAT! So just -ngl? Oh also do you know if it's possible to auto do 1 GPU then the next (ie sequential) - I have to manually set --device CUDA0 for smallish models, and probs distributing it amongst say all GPUs causes communication overhead!
Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @02:47AM (#43943687)
Ah no I mean we can omit the whole "-ngl N" argument for now, as it is internally set to -1 by default in CPP code (instead of being 0 traditionally), and -1 meaning offload everything to GPU

I have no idea how to specify custom layer specs with multi GPU, but that is interesting!

Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @02:57AM (#43943731)
WAIT so GPU offloading is on by DEFAULT? Oh my fantastic! For now I have to "guess" via a Python script - ie I sum sum up all the .gguf split files in filesize, then detect CUDA memory usage, and specify approximately how many GPUs ie --device CUDA0,CUDA1 etc
Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @03:07AM (#43943777)
Ahhh no sorry I forgot that the actual code controlling this is inside llama-model.cpp ; sorry for the misinfo, the -ngl only set to max by default if you're using Metal backend

(See the code in side llama_model_default_params())

Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @03:24AM (#43943846)
Oh no worries! I re-edited my comment to account for it :)