News for hackers. Stuff that matters.

A comment on Real-time LLM Inference on Standard GPUs: 3k tokens/s per request. Parent comment

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by kirtivr on Friday May 29, 2026 @06:56AM (#48321519)
They got 1K tok/s with Deepseek v4 Pro. That's kinda cool..
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @07:00AM (#48321550)
Thanks. To be fair, this number is what we expect to get once we port DeepSeek V4 in our engine on the upcoming generation of GPUs!
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gbnwl on Friday May 29, 2026 @04:01PM (#48328437)
No they didn't, they predict they'll get that much. Also worth noting the prediction assumes running at MXFP4/FP8 quantization.
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by dippatel1994 on Friday June 05, 2026 @02:16PM (#48416222)
Exactly! Any optimization for local inference is a welcome change IMHO!
[ Parent | on HN ]