News for hackers. Stuff that matters.
Re:Benchmark Framework Desktop Mainboard and 4-node cluster by zozbot234 on Friday August 08, 2025 @03:00AM (#44834304)
iGPUs (and NPUs) are not very useful for LLM inference, they only help somewhat in the prompt pre-processing phase. The CPU has worse bulk compute but far better access to system memory bandwidth, so it wins in token generation where that's the main factor.

My conspiracy theory is that it would help if contributors kept the Vulkan Compute proposed support up to date with new Ollama versions; no maintainer wants to deal with out-of-date pull req's.

There are no comments on this comment yet.