News for hackers. Stuff that matters.
Re:Open models by OpenAI by liuliu on Tuesday August 05, 2025 @02:16PM (#44802004)
Kimi K2 paper said that the model sparsity scales up with parameters pretty well (MoE sparsity scaling law, as they call, basically calling Llama 4 MoE "done wrong"). Hence K2 has 128:1 sparsity.
Re:Open models by OpenAI by throwdbaaway on Tuesday August 05, 2025 @06:40PM (#44805380)
I thought Kimi K2 uses 8 active experts out of 384? Sparsity should be 48:1. Indeed Llama4 Maverick is the only one that has 128:1 sparsity.
Re:Open models by OpenAI by liuliu on Wednesday August 06, 2025 @01:06PM (#44814694)
You are right. I mis-remembered the sparsity part of K2. The "done wrong" part I was thinking about how the scout -> maverick -> behemoth doesn't scale sparsity according to any formula (less sparse -> sparse -> less sparse).
Re:Open models by OpenAI by throwdbaaway on Thursday August 07, 2025 @12:19AM (#44820530)
> how the scout -> maverick -> behemoth doesn't scale sparsity according to any formula (less sparse -> sparse -> less sparse)

Ah I see. I didn't notice that behemoth has the same sparsity as scout. That seems quite random indeed.