News for hackers. Stuff that matters.

A comment on Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model. Parent comment

Re:Kimi K2 Thinking, a SOTA open-source trillion-parameter r... by oxcidized on Friday November 07, 2025 @03:42PM (#45850882)
> That's small enough to run well on ~$5,000 of hardware...

Honestly curious where you got this number. Unless you're talking about extremely small quants. Even just a Q4 quant gguf is ~130GB. Am I missing out on a relatively cheap way to run models well that are this large?

I suppose you might be referring to a Mac Studio, but (while I don't have one to be a primary source of information) it seems like there is some argument to be made on whether they run models "well"?

[ Parent | on HN ]
Re:Kimi K2 Thinking, a SOTA open-source trillion-parameter r... by simonw on Friday November 07, 2025 @04:24PM (#45851268)
Yes, I mean a Mac Studio with MLX.

An M3 Ultra with 256GB of RAM is $5599. That should just about be enough to fit MiniMax M2 at 8bit for MLX: https://huggingface.co/mlx-community/MiniMax-M2-8bit

Or maybe run a smaller quantized one to leave more memory for other apps!

Here are performance numbers for the 4bit MLX one: https://x.com/ivanfioravanti/status/1983590151910781298 - 30+ tokens per second.

[ Parent | on HN ]
Re:Kimi K2 Thinking, a SOTA open-source trillion-parameter r... by oxcidized on Friday November 07, 2025 @10:10PM (#45853783)
Thanks for the info! Definitely much better than I expected.
[ Parent | on HN ]
Re:Kimi K2 Thinking, a SOTA open-source trillion-parameter r... by zht on Saturday November 08, 2025 @08:11AM (#45856440)
It’s kinda misleading to omit the generally terrible prompt processing speed on Macs

30 tokens per second looks good until you have to wait minutes for the first token

[ Parent | on HN ]
Re:Kimi K2 Thinking, a SOTA open-source trillion-parameter r... by simonw on Saturday November 08, 2025 @08:28AM (#45856521)
The tweet I linked to includes that information in the chart.
[ Parent | on HN ]
Re:Kimi K2 Thinking, a SOTA open-source trillion-parameter r... by fzzzy on Friday November 07, 2025 @07:39PM (#45852983)
Running in cpu ram works fine. It’s not hard to build a machine with a terabyte of RAM.
[ Parent | on HN ]
Re:Kimi K2 Thinking, a SOTA open-source trillion-parameter r... by oxcidized on Friday November 07, 2025 @10:10PM (#45853778)
Admittedly I've not tried running on system RAM often, but every time I've tried it's been abysmally slow (< 1 T/s) when I've tried on something like KoboldCPP or ollama. Is there any particular method required to run them faster? Or is it just "get faster RAM"? I fully admit my DDR3 system has quite slow RAM...
[ Parent | on HN ]