News for hackers. Stuff that matters.
Re:RouteLLM: A framework for serving and evaluating LLM routers by Grimblewald on Wednesday July 10, 2024 @12:33AM (#40923730)
not true at all, you could have a efficient cheap model which is generally terrible at most things but has a savant like capacity for categorizing tasks by requirement and difficulty. Even easier when you dont need to support multiple languages and a truly staggering breadth of domains, like a conventional llm does. You could train a really small model to reject out of domain requests and partition the rest, running at a fraction of the cost of a more capable model.

There are no comments on this comment yet.