Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by sarthaksoni on Thursday August 07, 2025 @02:42AM
(#44821329)
Reading this made me realize how easy it is to set up GPT-OSS 20B in comparison. I had it running on my Mac in five minutes, thanks to Llama.
[ Parent
| on HN ]
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by DrPhish on Thursday August 07, 2025 @02:58AM
(#44821414)
Its also easy to do 120b on CPU if you have the resources. I had 120b running on my home LLM CPU inference box in just as long as it took to download the GGUFs, git pull and rebuild llama-server.
I had it running at 40t/s with zero effort and 50t/s with a brief tweaking.
Its just too bad that even the 120b isn't really worth running compared to the other models that are out there.
It really is amazing what ggerganov and the llama.cpp team have done to democratize LLMs for individuals that can't afford a massive GPU farm worth more than the average annual salary.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by wkat4242 on Thursday August 07, 2025 @03:48AM
(#44821727)
What hardware do you have? 50tk/s is really impressive for cpu.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by DrPhish on Thursday August 07, 2025 @04:45AM
(#44822141)
2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card.
I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by testaburger on Thursday August 07, 2025 @05:29AM
(#44822395)
Which specific model epcys? And if it's not too much to ask which motherboard and power supply? I'm really interested in building something similar
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by smartbit on Thursday August 07, 2025 @06:09AM
(#44822641)
Looking at https://news.ycombinator.com/submitted?id=DrPhish it's probably this machine https://rentry.co/miqumaxx
* Gigabyte MZ73-LM1 with two AMD EPYC GENOA 9334 QS 64c/128t
* 24 sticks of M321R4GA3BB6-CQK 32GB DDR5-4800 RDIMM PC5-38400R
* 24GB A5000
Note that the RAM price almost doubled since Jan 2024
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by wkat4242 on Thursday August 07, 2025 @05:05AM
(#44822256)
Wow nice!! That's a really good deal for that much hardware.
How many tokens/s do you get for DeepSeek-R1?
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by DrPhish on Thursday August 07, 2025 @10:29AM
(#44824925)
Thanks, it was a bit of a gamble at the time (lots of dodgy ebay parts), but it paid off.
R1 starts at about 10t/s on an empty context but quickly falls off. I'd say the majority of my tokens are generating around 6t/s.
Some of the other big MoE models can be quite a bit faster.
I'm mostly using QwenCoder 480b at Q8 these days for 9t/s average. I've found I get better real-world results out of it than K2, R1 or GLM4.5.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by ekianjo on Thursday August 07, 2025 @06:01AM
(#44822600)
thats a r/localllama user right there
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by fouc on Thursday August 07, 2025 @11:41AM
(#44825950)
I've seen some mentions of pure-cpu setups being successful for large models using old epyc/xeon workstations off ebay with 40+ cpus. Interesting approach!
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by SirMaster on Thursday August 07, 2025 @09:51AM
(#44824460)
I'm getting 20 tokens/sec on the 120B model with a 5060Ti 16GB and a regular desktop Ryzen 7800x3d with 64GB of DDR5-6000.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by wkat4242 on Thursday August 07, 2025 @04:20PM
(#44829846)
Wow that's not bad. It's strange, for me it is much much slower on a Radeon Pro VII (also 16GB, with a memory bandwidth of 1TB/s!) and a Ryzen 5 5600 with also 64GB. It's basically unworkably slow. Also, I only get 100% CPU when I check ollama ps, the GPU is not being used at all :( It's also counterproductive because the model is just too large for 64GB.
I wonder what makes it work so well on yours! My CPU isn't much slower and my GPU probably faster.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by magicalhippo on Thursday August 07, 2025 @05:29PM
(#44830612)
AMD basically decided they wanted to focus on HPC and data center customers rather than consumers, and so GPGPU driver support for consumer cards has been
non-existing or terrible[1].
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by wkat4242 on Saturday August 09, 2025 @06:43PM
(#44851043)
The Radeon VII Pro is not a consumer card though and works well with ROCm. It even has datacenter "grade" HBM2 memory that most Nvidias don't have. The continuing support has been dropped but ROCm of course still works fine. It's nearly as fast in Ollama as my 4090 (which I don't use for AI regularly but I just play with it sometimes)
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by exe34 on Thursday August 07, 2025 @04:42AM
(#44822114)
I imagine the gguf is quantised stuff?
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by DrPhish on Thursday August 07, 2025 @04:46AM
(#44822147)
No, I’m running the unquantized 120b
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by amelius on Thursday August 07, 2025 @05:14AM
(#44822301)
Why is it hard to set up llms? You can just ask an llm to do it for you, no? If this relatively simple task is already too much for llms then what good are they?
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by diggan on Thursday August 07, 2025 @05:34AM
(#44822445)
In the case of the GPT-OSS models, the worst (time consuming) part of supporting it is the new format they've trained the model with, "OpenAI harmony", in my own clients I couldn't just replace the model and call it a day, but still working on getting then to work correctly with tool calling...
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by CraigRood on Thursday August 07, 2025 @05:21AM
(#44822338)
I was playing with it yesterday and every single session gave me factually incorrect information.
Speed and ease of use is one thing, but it shouldn't be at the cost of accuracy.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by OliverGuy on Thursday August 07, 2025 @06:33AM
(#44822800)
If you are trying to get facts out of an LLM you are using it wrong, if you want a fact it should use a tool (eg we search, rag etc) to get the information that contains the fact (Wikipedia page, documentation etc) and then parse that document for the fact and return it to you.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by CraigRood on Sunday August 10, 2025 @02:09PM
(#44857061)
These tools are literally being marketed as AI, yet it presents false information as fact. 'using it wrong' can't be an argument here. I would rather then tool is honest about confidence levels and mechanisms to research further - then feed that fact back into 'AI' for the next step.