News for hackers. Stuff that matters.

A comment on You Should Probably Pay Attention to Tokenizers. Parent comment

Re:You Should Probably Pay Attention to Tokenizers by stephantul on Thursday October 24, 2024 @12:09AM (#41931816)
Not that I know of, but encoding orthography in a fixed width vector usually carries the assumption that words with the same prefix are more similar. So there’s an alignment problem. You usually solve this using dynamic programming, but that doesn’t work in a vector.

For example “parent” and “parents” are aligned, they share letters in the same position, but “skew” and “askew” share no letters in the same position.

[ Parent | on HN ]
Re:You Should Probably Pay Attention to Tokenizers by p1esk on Thursday October 24, 2024 @01:12AM (#41932137)
The other 500 values in the skew/askew vectors will be similar though. The 12 character values don’t need to be aligned, their function is to provide spelling. Adding such info will probably help LLM answer questions requiring character level knowledge (e.g. counting ‘r’s in ‘strawberry’).
[ Parent | on HN ]