Re:The Bitter Lesson is coming for Tokenization
by anonymoushn on Wednesday June 25, 2025 @12:15PM
(#44378970)
It's about the immutability of the network at runtime. But I really don't think this is a big deal. General-purpose computers are immutable after they are manufactured, but can exhibit a variety of useful behaviors when supplied with different data. Human intelligence also doesn't rely on designing and manufacturing revised layouts for the nervous system (within a single human's lifetime, for use by that single human) to adapt to different settings.
[ Parent
| on HN ]
Re:The Bitter Lesson is coming for Tokenization
by mgraczyk on Wednesday June 25, 2025 @01:21PM
(#44379734)
LLMs, considered as a function of input and output, are not immutable at runtime. They create tokens that change the function when it is called again. That breaks most theoretical arguments
Re:The Bitter Lesson is coming for Tokenization
by anonymoushn on Wednesday June 25, 2025 @02:20PM
(#44380342)
Sure. Another view is that an LLM is an immutable function from document-prefixes to next-token distributions.
Re:The Bitter Lesson is coming for Tokenization
by mgraczyk on Wednesday June 25, 2025 @02:24PM
(#44380391)
But that view is wrong, the model outputs multiple tokens.
The right alternative view is that it's an immutable function from prefixes to a distribution over all possible sequences of tokens less than (context_len - prefix_len).
There are no mutable functions that cannot be viewed as immutable in a similar way. Human brains are an immutable function from input sense-data to the combination (brain adaptation, output actions). Here "brain adaptation" doing a lot of work, but so would be "1e18 output tokens". There is much more information contained within the latter