Re:Show HN: Open-Source Load Balancer for Llama.cpp
by asne11 on Sunday June 02, 2024 @02:01AM
(#40551741)
"slot" is a processing unit. Either GPU or CPU. I believe `llama.c` is only CPU so I'm guessing 1 slot = 1 core (or thread)?
[ Parent
| on HN ]
Re:Show HN: Open-Source Load Balancer for Llama.cpp
by new299 on Sunday June 02, 2024 @02:58AM
(#40551980)
llama.cpp is not CPU only…
Re:Show HN: Open-Source Load Balancer for Llama.cpp
by kgeist on Sunday June 02, 2024 @05:14AM
(#40552669)
Llama.cpp can run on CPU, on GPU, or in mixed mode (some layers run on CPU and some on GPU if you don't have enough VRAM).
Re:Show HN: Open-Source Load Balancer for Llama.cpp
by mcharytoniuk on Sunday June 02, 2024 @05:54AM
(#40552836)
It divides the context into smaller "slots", so it can process requests concurrently with continuous batching. See also: https://github.com/ggerganov/llama.cpp/tree/master/examples/...