Re:Meta Llama 3
by staticman2 on Monday April 22, 2024 @09:08AM
(#40114000)
"For example 8x7B Mixtral only executes 13B parameters per token, but it behaves comparable to 34B and 70B models"
Are you sure about that? I'm pretty sure Miqu (the leaked Mistral 70b model) is generally thought to be smarter than Mixtral 8x7b.
Similarly, if GPT-4 is really 1.8T you would expect it to produce output of similar quality to a comparable 1.8T model without MoE architecture.