News for hackers. Stuff that matters.

A comment on Real-time LLM Inference on Standard GPUs: 3k tokens/s per request.

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by irishcoffee on Friday May 29, 2026 @07:03AM (#48321581)
NVIDIA H200 Is not a standard GPU. 8 of them in a box with a cpu and ram costs close to the same as a house.

I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading.

I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development.

If you can’t connect a monitor, it isn’t a standard GPU, at least not in the way people have spoken about GPUs until a few years ago.

[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @07:05AM (#48321594)
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.

Sorry for the confusion

[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by embedding-shape on Friday May 29, 2026 @07:27AM (#48321757)
Do you think maybe changing your articles title from "Real-time LLM Inference on Standard GPUs" to "Real-time LLM Inference on Standard Datacenter GPUs" might make sense here? Given more people seem confused by the title than not, and you could clear this up relatively easily, at least on your website although might be late to fix the HN title.
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @07:37AM (#48321846)
YES - I just updated the title of our article according to your suggestion.
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by irishcoffee on Friday May 29, 2026 @07:35AM (#48321831)
Oh, it isn't confusing, it is misleading. A standard GPU lets you connect a monitor. A datacenter GPU lets you do headless math.
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @07:38AM (#48321851)
I updated the article title accordingly
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by bcjdjsndon on Friday May 29, 2026 @10:15AM (#48323395)
Standard != Datacentre
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by dandanua on Saturday May 30, 2026 @11:13AM (#48337118)
This blog post clearly targets VCs, but what they are doing is legit and can improve the performance of local models on low-end hardware as well, especially since their priority is to optimize non-batched inference.
[ Parent | on HN ]