News for hackers. Stuff that matters.
Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by qeternity on Wednesday April 24, 2024 @06:49PM (#40150815)
"Expert" in MoE has no bearing on what you might think of as a human expert.

It's not like there is one expert that is proficient at science, and one that is proficient in history.

For a given inference request, you're likely to activate all the experts at various points. But for each individual forward pass (e.g. each token), you are only activating a few.

There are no comments on this comment yet.