News for hackers. Stuff that matters.

A comment on The Bitter Lesson is coming for Tokenization. Parent comment

Re:The Bitter Lesson is coming for Tokenization by danielmarkbruce on Tuesday June 24, 2025 @06:28PM (#44371710)
Assuming decent data, it won't be stochastic sampling for many math operations/input combinations. When people suggest LLMs with tokenization could learn math, they aren't suggesting a small undertrained model trained on crappy data.
[ Parent | on HN ]
Re:The Bitter Lesson is coming for Tokenization by anonymoushn on Tuesday June 24, 2025 @07:41PM (#44372243)
I mean, this depends on your sampler. With temp=1 and sampling from the raw output distribution, setting aside numerics issues, these models output nonzero probability of every token at each position
[ Parent | on HN ]
Re:The Bitter Lesson is coming for Tokenization by danielmarkbruce on Wednesday June 25, 2025 @01:58PM (#44380103)
A large model well trained on good data will have logits so negative for something like "1+1=" -> 3 that they won't come up in practice unless you sample in a way to deliberately misuse the model.
[ Parent | on HN ]