News for hackers. Stuff that matters.

A comment on Real-time LLM Inference on Standard GPUs: 3k tokens/s per request.

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by freediddy on Friday May 29, 2026 @02:45PM (#48327565)
Who cares about token speed? What is the quality of the results like? I don't know why people are so fixated on token speed, since no one cares how quickly it can spew garbage. Most reasonable people are happier waiting a bit more for accurate results.
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by esafak on Friday May 29, 2026 @03:35PM (#48328149)
It matters on consumer hardware since barely any model runs at reasonable speed.
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @03:55PM (#48328363)
It also matters for thinking models and for agentic workflows, especially in software engineering, where a lot of tokens need to be output in iterative loops before the user sees any result.

This is our main use case.

[ Parent | on HN ]