News for hackers. Stuff that matters.
Re:Generating one token at a time is a blessing in disguise by halflings on Sunday March 29, 2026 @04:49PM (#47567161)
LLMs generate their output one token at a time. The first thought when you learn this is that this is a huge performance bottleneck, as we are used to highly parallelized systems.

However, a large part of what makes LLMs feel so magical comes from this bottleneck.