Show HN: Sturnus, a latency-based LLM load balancer
https://github.com/sturnus-dev/sturnus
I built sturnus because I wanted the latency-based load balancing of an AI gateway but without a big framework selling an enterprise plan.It’s a single static binary that does one thing, aiming to avoid the large surface area and security issues that have plagued some LLM gateways.Traffic is distributed by LLM provider latency and availability. The share going to other providers acts as a...