mcharytoniuk
Ask HN: Async Python Adoption?
Are there any studies, large-scale polls, or anything about async coding adoption in Python?I wonder how widely the community accepts it, how widespread its usage is, and what the general sentiment is towards it.
Re:Show HN: Open-Source Load Balancer for Llama.cpp
on Monday June 03, 2024 @07:06AM (#40561453)
Yes, exactly. You can split the available context into "slots" (chunks) so it can handle multipe requests concurrently. The number of them is configurable.
Re:Show HN: Open-Source Load Balancer for Llama.cpp
on Sunday June 02, 2024 @05:54AM (#40552836)
It divides the context into smaller "slots", so it can process requests concurrently with continuous batching. See also: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
Re:Show HN: Open-Source Load Balancer for Llama.cpp
on Sunday June 02, 2024 @05:53AM (#40552833)
Just open an issue if you need anything. I want to make it as good and helpful as possible. Every kind of feedback is appreciated.
Re:Show HN: Open-Source Load Balancer for Llama.cpp
on Sunday June 02, 2024 @05:52AM (#40552830)
Currently, it is a single instance in memory, so it doesn't transfer state. HA is on the roadmap; only then will it need some kind of distributed state store.
Local states are reported by the agents installed alongside llama.cpp to the load balancer. That means they can be dynamically added and removed; it doesn't need a central configuration.
Re:Show HN: Open-Source Load Balancer for Llama.cpp
on Sunday June 02, 2024 @05:50AM (#40552820)
In progress. I added that to the readme; I need the feature myself. :)
Show HN: Open-Source Load Balancer for Llama.cpp
https://github.com/distantmagic/paddler
Stateful load balancer customized for llama.cpp (with a reverse proxy).
Show HN: Language-agnostic structured data extractor
https://github.com/distantmagic/structured
Started as an unofficial port of Instructor, but its language-agnostic and focused primarily on llama.cpp.Starts HTTP API server. You send JSON-Schema + Data, it responds with a JSON matching your schema, extracted from the data.
Show HN: Resonance – PHP Framework That Solves Real-Life Issues
https://github.com/distantmagic/resonance
Hello HN!I made a PHP framework with AI capabilities.The most important features: - Built on top of Swoole extension to support asynchronous features. - Opinionated, all-in-one solution for building web applications, particularly SaaS (it is not a boilerplate, though). - Integrates with llama.cpp to serve responses from open-source LLMs. - Uses minimal configuration and extensively relies on...