Re:Show HN: Open-Source Load Balancer for Llama.cpp
by mcharytoniuk on Monday June 03, 2024 @07:06AM
(#40561453)
Yes, exactly. You can split the available context into "slots" (chunks) so it can handle multipe requests concurrently. The number of them is configurable.
[ Parent
| on HN ]
There are no comments on this comment yet.