News for hackers. Stuff that matters.
Re:How the BPE tokenization algorithm used by large language... by cubefox on Thursday July 06, 2023 @04:03PM (#36621985)
The way I think about it: A token can be one to many bytes long, so they can be longer or shorter than a single character.

There are no comments on this comment yet.