Re:GPT-4 Details Leaked
by why_only_15 on Tuesday July 11, 2023 @03:59AM
(#36677811)
This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
[ Parent
| on HN ]
Re:GPT-4 Details Leaked
by ShamelessC on Tuesday July 11, 2023 @04:33AM
(#36678043)
Oh I’m happy to admit if I’m wrong in the details. My bad.
So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?