Re:Ask HN: Who uses open LLMs and coding assistants locally?...
by whatreason on Friday October 31, 2025 @10:02PM
(#45778679)
What do you use to run gpt-oss here? ollama, vLLM, etc
[ Parent
| on HN ]
Re:Ask HN: Who uses open LLMs and coding assistants locally?...
by embedding-shape on Saturday November 01, 2025 @08:09AM
(#45781029)
Not parent, but frequent user of GPT-OSS, tried all different ways of running it. Choice goes something like this:
- Need batching + highest total throughoutput? vLLM, complicated to deploy and install though, need special versions for top performance with GPT-OSS - Easiest to manage + fast enough: llama.cpp, easier to deploy as well (just a binary) and super fast, getting ~260 tok/s on a RTX Pro 6000 for the 20B version - Easiest for people not used to running shell commands or need a GUI and don't care much for performance: Ollama
Then if you really wanna go fast, try to get TensorRT running on your setup, and I think that's pretty much the fastest GPT-OSS can go currently.