Re:Generating one token at a time is a blessing in disguise
by halflings on Sunday March 29, 2026 @04:49PM
(#47567161)
LLMs generate their output one token at a time. The first thought when you learn this is that this is a huge performance bottleneck, as we are used to highly parallelized systems.
However, a large part of what makes LLMs feel so magical comes from this bottleneck.
However, a large part of what makes LLMs feel so magical comes from this bottleneck.