News for hackers. Stuff that matters.

A comment on Real-time LLM Inference on Standard GPUs: 3k tokens/s per request. Parent comment

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by dr_kiszonka on Friday May 29, 2026 @01:15PM (#48326220)
Thank you for explaining. Do you think there are still opportunities for stack optimizations to meaningfully speed up inference on single consumer-grade GPUs?
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @02:13PM (#48327060)
I'm sure there are, and I really hope we can work on consumer-grade GPUs at some point.

It should be possible to apply the same methodology (digging deep into the hardware details to understand all its little characteristics, and rethinking the inference stack around that).

[ Parent | on HN ]