News for hackers. Stuff that matters.
Re:Meta Llama 3 by chessgecko on Thursday April 18, 2024 @01:20PM (#40078536)
I think its almost certainly using at least two experts per token. It helps a lot during training to have two experts to contrast when putting losses on the expert router.

There are no comments on this comment yet.