Show HN: Open-Source Load Balancer for Llama.cpp

Re:Show HN: Open-Source Load Balancer for Llama.cpp by anonzzzies on Sunday June 02, 2024 @02:30AM (#40551886)

Does it do queuing ? Didn’t see it in the readme. I haven’t seen (but that says nothing at all) an open source solution that queues when all are busy and allow me to show a countdown of people in the queue. Like the closed ones do.

[ Parent | on HN ]

Re:Show HN: Open-Source Load Balancer for Llama.cpp by ritonlajoie on Sunday June 02, 2024 @03:12AM (#40552053)

I stumbled on this yesterday, it seems to have a queue concept.. https://github.com/ParisNeo/ollama_proxy_server

[ Parent | on HN ]

Re:Show HN: Open-Source Load Balancer for Llama.cpp by mcharytoniuk on Sunday June 02, 2024 @05:50AM (#40552820)

In progress. I added that to the readme; I need the feature myself. :)

[ Parent | on HN ]

Re:Show HN: Open-Source Load Balancer for Llama.cpp by friendly_chap on Sunday June 02, 2024 @04:10PM (#40556892)

I have actually open sourced one recently which supports queues - its both a desktop app and a daemon: https://github.com/singulatron/singulatron

[ Parent | on HN ]

Re:Show HN: Open-Source Load Balancer for Llama.cpp by 3abiton on Sunday June 02, 2024 @05:08PM (#40557210)

So many interesting projects, I just wish AI hardware was readily available.

[ Parent | on HN ]