News for hackers. Stuff that matters.
Re:GPT-4 Details Leaked by why_only_15 on Tuesday July 11, 2023 @03:59AM (#36677811)
This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
Re:GPT-4 Details Leaked by ShamelessC on Tuesday July 11, 2023 @04:33AM (#36678043)
Oh I’m happy to admit if I’m wrong in the details. My bad.

So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?