Re:Open models by OpenAI
by liuliu on Tuesday August 05, 2025 @02:16PM
(#44802004)
Kimi K2 paper said that the model sparsity scales up with parameters pretty well (MoE sparsity scaling law, as they call, basically calling Llama 4 MoE "done wrong"). Hence K2 has 128:1 sparsity.
[ Parent
| on HN ]
Re:Open models by OpenAI
by throwdbaaway on Tuesday August 05, 2025 @06:40PM
(#44805380)
I thought Kimi K2 uses 8 active experts out of 384? Sparsity should be 48:1. Indeed Llama4 Maverick is the only one that has 128:1 sparsity.
You are right. I mis-remembered the sparsity part of K2. The "done wrong" part I was thinking about how the scout -> maverick -> behemoth doesn't scale sparsity according to any formula (less sparse -> sparse -> less sparse).
Re:Open models by OpenAI
by throwdbaaway on Thursday August 07, 2025 @12:19AM
(#44820530)
> how the scout -> maverick -> behemoth doesn't scale sparsity according to any formula (less sparse -> sparse -> less sparse)
Ah I see. I didn't notice that behemoth has the same sparsity as scout. That seems quite random indeed.