Snowflake Arctic Instruct (128x3B MoE), largest open source model

Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by giantrobot on Wednesday April 24, 2024 @02:19PM (#40147820)

I believe the main draw of the MoE model is they don't all need to be in memory at once. They can be swapped based on context. In aggregate you get the performance of a much larger model (384b tokens) while using much less memory than such a model would require. If you had enough memory it could all be loaded but it doesn't need to be.

[ Parent | on HN ]

Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by Manabu-eo on Wednesday April 24, 2024 @03:21PM (#40148435)

Wrong. MoE models like this one usually chose a different and unpredictable mix of experts for each token, and as such you need all parameters at memory at once.

It lessens the number of parameters that need to be moved from memory to compute chip for each token, not from disk to memory.

[ Parent | on HN ]

Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by sp332 on Wednesday April 24, 2024 @03:18PM (#40148407)

Technically you could, but it would take much longer to do all that swapping.

[ Parent | on HN ]

Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by qeternity on Wednesday April 24, 2024 @06:49PM (#40150815)

"Expert" in MoE has no bearing on what you might think of as a human expert.

It's not like there is one expert that is proficient at science, and one that is proficient in history.

For a given inference request, you're likely to activate all the experts at various points. But for each individual forward pass (e.g. each token), you are only activating a few.

[ Parent | on HN ]