Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model
by hbrundage on Thursday August 15, 2024 @05:39PM
(#41260648)
Isn't 63% => 54% regression on MMLU-Pro a huge issue? They said that it excels at advanced reasoning but that seems like a big drawback there.
[ Parent
| on HN ]
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model
by kainan-ai on Thursday August 15, 2024 @06:19PM
(#41261069)
Yeah it doesn't win in every category. I will say watching it in the discord I saw its performance vary widely so the context and sys prompt plays a huge role. Initially it did great and solved some pretty heavy logic questions but after the context was loaded with trolling it degraded quite a bit and couldn't solve problems it previously was able to.