News for hackers. Stuff that matters.

A comment on Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020). Parent comment

Re:Tokenization for language modeling: BPE vs. Unigram Langu... by mcyc on Saturday May 31, 2025 @01:54PM (#44145875)
You can cross whitespace boundaries by setting flag `--split-on-whitespace` to false (it's true by default).

https://github.com/google/sentencepiece/blob/master/doc/opti...

[ Parent | on HN ]

There are no comments on this comment yet.