News for hackers. Stuff that matters.
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @05:24PM (#48329489)
True, and for third-party models we'll just re-use their public open weights.

There is a time-consuming part, though, that is performed manually by our (human) team: implement the logic of the model in C++ and assembly code in a super-optimized way, co-designed for each specific hardware card.

This can take months.

We hope to accelerate the process with AI agents, but we're not there yet.

There are no comments on this comment yet.