News for hackers. Stuff that matters.

Ask HN: LLM Router Recommendations

Basically I have a use-case where I am building an llm-based tutor where some questions you ask the tutor can be adequately answered by gpt-4o-mini but others required gpt-4o. I'm looking for a router that can route to gpt-4o only when necessary and otherwise route to the cheaper gpt-4o-mini. First of all, is this even something that can be done reliably at this point? If so, are there any recommendations for LLM routers that can do this?
( 4 comments | on HN )
Re:Ask HN: LLM Router Recommendations by phren0logy on Tuesday October 01, 2024 @08:13PM (#41715871)
The best option I have seen for routing is LiteLLM. Worth a look.
Re:Ask HN: LLM Router Recommendations by LifeOverIP on Tuesday October 01, 2024 @08:48PM (#41716090)
I would check out if Not Diamond (no personal affiliation) [0] fits your use case. They had a really interesting blog post recently where they open sourced their older model router [1].

[0] https://www.notdiamond.ai/features

[1] Blog - Routing on Random Forests - https://www.notdiamond.ai/blog/rorf

Re:Ask HN: LLM Router Recommendations by gbertb on Wednesday October 02, 2024 @01:11AM (#41717380)
Re:Ask HN: LLM Router Recommendations by iknownthing on Wednesday October 02, 2024 @11:01AM (#41721324)
That video looks like what I need, I'll give it a try, thanks