Re:Meta Llama 3
by chessgecko on Thursday April 18, 2024 @01:20PM
(#40078536)
I think its almost certainly using at least two experts per token. It helps a lot during training to have two experts to contrast when putting losses on the expert router.
[ Parent
| on HN ]
There are no comments on this comment yet.