News for hackers. Stuff that matters.

A comment on GPT-4o mini: advancing cost-efficient intelligence. Parent comment

Re:GPT-4o mini: advancing cost-efficient intelligence by laborcontract on Thursday July 18, 2024 @02:32PM (#40998428)
Sounds like you're describing mixture of experts, the architecture being used in openai's gpt-4 and mistral's mixtral series of models.
[ Parent | on HN ]
Re:GPT-4o mini: advancing cost-efficient intelligence by pants2 on Thursday July 18, 2024 @02:56PM (#40998649)
Not really, MoE is trained all at once and the 'experts' don't have pre-defined specializations. They end up being more like "punctuation expert" and "pronoun expert" than "math expert" and "french expert"
[ Parent | on HN ]
Re:GPT-4o mini: advancing cost-efficient intelligence by laborcontract on Thursday July 18, 2024 @03:06PM (#40998739)
Haven't tried any yet, but it sounds like parent may be interested in an LLM router. https://github.com/lm-sys/RouteLLM
[ Parent | on HN ]