News for hackers. Stuff that matters.

A comment on Show HN: Open-Source Load Balancer for Llama.cpp. Parent comment

Re:Show HN: Open-Source Load Balancer for Llama.cpp by tjfontaine on Sunday June 02, 2024 @01:12AM (#40551543)
Looks like it’s to only have one completion running at a time currently, https://github.com/distantmagic/paddler/blob/main/loadbalanc... I was curious if there was any other cache tracking happening
[ Parent | on HN ]
Re:Show HN: Open-Source Load Balancer for Llama.cpp by mcharytoniuk on Sunday June 02, 2024 @05:53AM (#40552833)
Just open an issue if you need anything. I want to make it as good and helpful as possible. Every kind of feedback is appreciated.
[ Parent | on HN ]