Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by blindr on Friday May 29, 2026 @01:44PM (#48326629)

For me it's 3.4k tok/s of pure nonsense, the model is bad, you tell it it's wrong, it acknowledge it's wrong and repeats the same nonsense. It reminds me my nephew though. Ask it something like: "I want to play the guitar on the surface of the Moon. What speakers do you suggest." and then "But Moon has no atmosphere, how the sound will travel?".

[ Parent | on HN ]

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @02:10PM (#48327030)

Note that this coding model is trained on programming use cases, and is also not tuned for multi-turn chat.

You can ask it to implement an algorithm; we provide suggested prompts you can test.

Also, this tech preview is really about the speed of the inference engine (not the model itself) so I'm glad you got 3.4k tok/s!

[ Parent | on HN ]

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by blindr on Saturday May 30, 2026 @07:04AM (#48334923)

That's what I tested first and the model failed, even after suggestions that it was wrong and how to fix it's errors.

[ Parent | on HN ]