News for hackers. Stuff that matters.
Re:Clef: Open-source decision models, and new RL fine-tuning... by Twirrim on Thursday October 01, 2026 @05:36PM (#49927347)
We already have examples of LLMs running 16k+ tokens a second using custom ASICs.

It's down at the moment (Not sure if it'll return?) but Chat Jimmy[0] produced by Taalas[1] was powered by an ASIC running Llama 3.1 8B, and hitting 17,000 tokens/sec. It was amazing to use, you'd no sooner have hit enter than you had a full response back. I actually found its speed to be a problem for interactive stuff, as every answer got several paragraphs I'd then wade through, vs a model populating text closer to my reading speed.

I appreciate, there are differences between an 8 billion parameter model and something GPT-4-ish, but we're currently in the middle of a race between a half dozen or so companies to produce the next best frontier model, which requires their infrastructure to be dynamic.

We really don't always need newer better faster stronger models, there's quite a lot of room for "good enough" where getting 17kt/s at significantly lower power would be amazing.

[0] https://chatjimmy.ai/ [1] https://taalas.com/

Re:Clef: Open-source decision models, and new RL fine-tuning... by AshamedBadger56 on Thursday October 01, 2026 @05:52PM (#49927500)
>I actually found its speed to be a problem for interactive stuff, as every answer got several paragraphs I'd then wade through, vs a model populating text closer to my reading speed.

It would be interesting to pair the super fast model with a normal speed model. Have the super fast one do all the background research, code writing, etc. The normal model would just relay the needed info to you at a more reasonable pace.

Re:Clef: Open-source decision models, and new RL fine-tuning... by drewstiff on Friday October 02, 2026 @06:38AM (#49931983)
Why do you even need a second model for that? There are many simpler ways to slow down the rate that the near-immediate response is pseudo-typed onto the screen.
Re:Clef: Open-source decision models, and new RL fine-tuning... by Twirrim on Tuesday October 06, 2026 @01:47AM (#49974680)
Agreed, could be something as dumb as a simple javascript snippet running in the browser. Would be nuts to use something like an LLM just for that.