News for hackers. Stuff that matters.

A comment on Real-time LLM Inference on Standard GPUs: 3k tokens/s per request. Parent comment

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by __natty__ on Friday May 29, 2026 @12:04PM (#48325027)
As for the demo it's fast and extremely dumb like expected for 2B. I asked how to stop drinking habit and in just one follow-up message it recommended trying 8% ABV. Hilarious.
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @12:06PM (#48325062)
it's also a coding model
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by nethi on Friday May 29, 2026 @12:36PM (#48325586)
Nah. it says it can't even write python code
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by averne_ on Friday May 29, 2026 @12:47PM (#48325747)
I tried with some simple prompts (fibonacci, linked list manipulation) and it worked nicely.
[ Parent | on HN ]