News for hackers. Stuff that matters.
Re:Show HN: Open-Source Load Balancer for Llama.cpp by mcharytoniuk on Monday June 03, 2024 @07:06AM (#40561453)
Yes, exactly. You can split the available context into "slots" (chunks) so it can handle multipe requests concurrently. The number of them is configurable.

There are no comments on this comment yet.