News for hackers. Stuff that matters.
Re:How the BPE tokenization algorithm used by large language... by weinzierl on Thursday July 06, 2023 @12:59PM (#36619054)
Ok, I understand. That helped, thanks.

And also suddenly the B in BPE makes a lot of sense.

Re:How the BPE tokenization algorithm used by large language... by cubefox on Thursday July 06, 2023 @04:03PM (#36621985)
The way I think about it: A token can be one to many bytes long, so they can be longer or shorter than a single character.