News for hackers. Stuff that matters.

A comment on Local LLM inference – impressive but too hard to work with. Parent comment

Re:Local LLM inference – impressive but too hard to work with by Havoc on Monday April 21, 2025 @08:45PM (#43758050)
Apparently for the bigger model checking a token is faster than generating a fresh one. So if they tiny model gets it right you get a tiny speed bump. Can’t say I fully understand it either why it’s faster to check

Needs a pretty large difference in size to result in a speedup. 0.5 vs 27b is the only ones I’ve seen a speedbump

[ Parent | on HN ]

There are no comments on this comment yet.