News for hackers. Stuff that matters.

A comment on Kimi-K3 Releases on HuggingFace 7/27. Parent comment

Re:Kimi-K3 Releases on HuggingFace 7/27 by johndough on Monday July 27, 2026 @11:22AM (#49070973)
Update: Looks like the model is larger after all (1561.44 GB). Only the MoE weights are MXFP4, while the other weights are BF16 (and a few FP32).

* Sparse Experts: 1481.4 GB

* Dense Experts: 1.9 GB

* Self-Attention: 72.4 GB

* LLM Head: 2.4 GB

* Embeddings: 2.4 GB

* Vision Encoder: 0.35 GB (surprisingly small)

plus some miscellaneous parameters.

Most importantly, we now know that the model has 104B active parameters, which is quite a lot.

[ Parent | on HN ]

There are no comments on this comment yet.