News for hackers. Stuff that matters.

A comment on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020). Parent comment

Re:Tokenization for language modeling: BPE vs. Unigram Langu... by cschmidt on Saturday May 31, 2025 @08:59AM (#44143975)
It appears to be the top n-grams scored by the product of frequency and length. Including the frequency weighting is a bit nonstandard among ablative methods.

See line 233: https://github.com/google/sentencepiece/blob/master/src/unig...

I would suspect the n-gram counts don't cross pre-token boundaries, but I don't have time to find that in the code right now.

[ Parent | on HN ]
Re:Tokenization for language modeling: BPE vs. Unigram Langu... by mcyc on Saturday May 31, 2025 @01:54PM (#44145875)
You can cross whitespace boundaries by setting flag `--split-on-whitespace` to false (it's true by default).

https://github.com/google/sentencepiece/blob/master/doc/opti...

[ Parent | on HN ]
Re:Tokenization for language modeling: BPE vs. Unigram Langu... by cschmidt on Sunday June 01, 2025 @10:14AM (#44151048)
Anyone reading this in the future, I meant to say the length weighting is a bit nonstandard. It is usually by frequency. Oops
[ Parent | on HN ]