Vision Now Available in Llama.cpp

Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @02:47AM (#43943687)

Ah no I mean we can omit the whole "-ngl N" argument for now, as it is internally set to -1 by default in CPP code (instead of being 0 traditionally), and -1 meaning offload everything to GPU

I have no idea how to specify custom layer specs with multi GPU, but that is interesting!

[ Parent | on HN ]

Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @02:57AM (#43943731)

WAIT so GPU offloading is on by DEFAULT? Oh my fantastic! For now I have to "guess" via a Python script - ie I sum sum up all the .gguf split files in filesize, then detect CUDA memory usage, and specify approximately how many GPUs ie --device CUDA0,CUDA1 etc

[ Parent | on HN ]

Re:Vision Now Available in Llama.cpp by ngxson on Saturday May 10, 2025 @03:07AM (#43943777)

Ahhh no sorry I forgot that the actual code controlling this is inside llama-model.cpp ; sorry for the misinfo, the -ngl only set to max by default if you're using Metal backend

(See the code in side llama_model_default_params())

[ Parent | on HN ]

Re:Vision Now Available in Llama.cpp by danielhanchen on Saturday May 10, 2025 @03:24AM (#43943846)

Oh no worries! I re-edited my comment to account for it :)

[ Parent | on HN ]