Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour...
by qeternity on Wednesday April 24, 2024 @06:49PM
(#40150815)
"Expert" in MoE has no bearing on what you might think of as a human expert.
It's not like there is one expert that is proficient at science, and one that is proficient in history.
For a given inference request, you're likely to activate all the experts at various points. But for each individual forward pass (e.g. each token), you are only activating a few.
It's not like there is one expert that is proficient at science, and one that is proficient in history.
For a given inference request, you're likely to activate all the experts at various points. But for each individual forward pass (e.g. each token), you are only activating a few.