Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by ilaksh on Friday May 29, 2026 @06:36AM (#48321383)

Could be amazing, but it's hard to judge if it will really work with say a 27 B model or larger. We can already get pretty good speed with a 2B model.

[ Parent | on HN ]

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @07:03AM (#48321575)

thanks! we explain how it scales to larger models in the last section the OP blog post

[ Parent | on HN ]

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by bcjdjsndon on Friday May 29, 2026 @10:14AM (#48323374)

Shame you stopped short of actually benchmarking that scale though, eh?

[ Parent | on HN ]

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @11:10AM (#48324079)

will do - we are a small team and it takes time to implement and optimize a new model, whatever the size.

[ Parent | on HN ]

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by lostmsu on Friday May 29, 2026 @05:03PM (#48329227)

You don't even need to train the model just to see if you can infer it at the claimed speed

[ Parent | on HN ]

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @05:24PM (#48329489)

True, and for third-party models we'll just re-use their public open weights.

There is a time-consuming part, though, that is performed manually by our (human) team: implement the logic of the model in C++ and assembly code in a super-optimized way, co-designed for each specific hardware card.

This can take months.

We hope to accelerate the process with AI agents, but we're not there yet.

[ Parent | on HN ]

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by bcjdjsndon on Friday May 29, 2026 @05:10PM (#48329319)

[ Parent | on HN ]