News for hackers. Stuff that matters.
Re:Benchmark Framework Desktop Mainboard and 4-node cluster by wmf on Thursday August 07, 2025 @05:14PM (#44830454)
If you think the future is small models (27B) get Nvidia; if you think larger models (70-120B) are worth it then you need AMD or Apple.
Re:Benchmark Framework Desktop Mainboard and 4-node cluster by yencabulator on Thursday August 07, 2025 @05:28PM (#44830604)
I wonder how much MoE will disrupt this. qwen3:30b-a3b is pretty good even on pure CPU, but a lot smarter than a 3B parameter model. If the CPU-GPU bottleneck isn't too tight, a large model might be able to sustainably cache the currently active experts in GPU RAM.
Re:Benchmark Framework Desktop Mainboard and 4-node cluster by hengheng on Thursday August 07, 2025 @07:14PM (#44831548)
The recent qwen3 models run fine on CPU + GPU, and so does gpt-oss. LM Studio and Ollama are turnkey solutions where the user has to know nothing about memory management. But finding benchmarks for these hybrid setups is astonishingly difficult.

I keep thinking that the bottleneck has to be CPU RAM, and for a large model the difference would be minor. For example with an 100 GByte model such as quantised gpt-oss-120B, I imagine that going from 10G to 24G would scale up my tk/s like 1/90 -> 1/76, so 20% advantage? But I can't find much on the high-level scaling math. People seem to either create calculators that oversimplify, or they seem too deep into the weeds.

I'd like a new anandtech please.

Re:Benchmark Framework Desktop Mainboard and 4-node cluster by whizzter on Friday August 08, 2025 @03:50AM (#44834596)
Doesn't matter, people will always find ways to eat RAM despite finding more clever ways to do things.
Re:Benchmark Framework Desktop Mainboard and 4-node cluster by yencabulator on Friday August 08, 2025 @11:27AM (#44838171)
MoE eats the same amount of RAM but accesses less of it.
Re:Benchmark Framework Desktop Mainboard and 4-node cluster by sliken on Saturday August 09, 2025 @12:49AM (#44844103)
More accurately, same amount of ram but accessed in a cache friendly manner with greater locality.