News for hackers. Stuff that matters.

A comment on Meta Llama 3. Parent comment

Re:Meta Llama 3 by vacuumcl on Thursday April 18, 2024 @12:36PM (#40078032)
Comparing to the numbers here https://www.anthropic.com/news/claude-3-family the ones of Llama 400B seem slightly lower, but of course it's just a checkpoint that they benchmarked and they are still training further.
[ Parent | on HN ]
Re:Meta Llama 3 by causal on Thursday April 18, 2024 @01:08PM (#40078416)
Indeed. But if GPT-4 is actually 1.76T as rumored, an open-weight 400B is quite the achievement even if it's only just competitive.
[ Parent | on HN ]
Re:Meta Llama 3 by cjbprime on Thursday April 18, 2024 @01:41PM (#40078719)
The rumor is that it's a mixture of experts model, which can't be compared directly on parameter count like this because most weights are unused by most inference passes. (So, it's possible that 400B non-MoE is the same approximate "strength" as 1.8T MoE in general.)
[ Parent | on HN ]