News for hackers. Stuff that matters.
Re:RouteLLM: A framework for serving and evaluating LLM routers by antupis on Thursday July 11, 2024 @02:07AM (#40934043)
Cost is a plus but at least what I see is that getting good response time is even bigger. Something like OpenAI Azure instances are inconsistent and it is far too normal to get a 40sec lag with responses with gpt4-o.

There are no comments on this comment yet.