News for hackers. Stuff that matters.

A comment on Real-time LLM Inference on Standard GPUs: 3k tokens/s per request.

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by DeathArrow on Friday May 29, 2026 @01:33PM (#48326452)
>This preview runs a 2B model

I guess with 1B or 500M model inference would be even faster?

[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @02:55PM (#48327687)
In theory yes, although not in a linearly proportional way, because in practice our memory streaming is not yet perfect. There are still some fixed costs that we did not fully optimize (for now).
[ Parent | on HN ]