Re:RouteLLM: A framework for serving and evaluating LLM routers
by antupis on Thursday July 11, 2024 @02:07AM
(#40934043)
Cost is a plus but at least what I see is that getting good response time is even bigger. Something like OpenAI Azure instances are inconsistent and it is far too normal to get a 40sec lag with responses with gpt4-o.
[ Parent
| on HN ]
There are no comments on this comment yet.