News for hackers. Stuff that matters.
Re:How the BPE tokenization algorithm used by large language... by space_fountain on Thursday July 06, 2023 @12:15PM (#36618282)
Yes, but the tokens are translated into bytes not characters. There are only 256 distinct bytes so GPT models can easily be trained to produce any character. Probably the problem will be how sensible or understandable the binary form of Chinese characters in Unicode are, but that will be a problem for the model, not the tokenizer
Re:How the BPE tokenization algorithm used by large language... by weinzierl on Thursday July 06, 2023 @12:59PM (#36619054)
Ok, I understand. That helped, thanks.

And also suddenly the B in BPE makes a lot of sense.

Re:How the BPE tokenization algorithm used by large language... by cubefox on Thursday July 06, 2023 @04:03PM (#36621985)
The way I think about it: A token can be one to many bytes long, so they can be longer or shorter than a single character.