News for hackers. Stuff that matters.

Service to auto route LLM/Model traffic

I suggest creating a service that monitors all the LLMs/AI model deployment services on Azure, AWS, Google Cloud, Groq, Krutrim, and other cloud deployments and routes your traffic based on availability, latency, rate-limiting, and other parameters. Manage everything at the backend, just pay a single unified bill, with no deployments or accounts on any other service. Sounds cool?
( 1 comment | on HN )
Re:Service to auto route LLM/Model traffic by yorwba on Wednesday February 19, 2025 @03:32AM (#43099930)
Sounds similar to SkyPilot: https://docs.skypilot.co

But you still need to create your own accounts for each service.