News for hackers. Stuff that matters.
Re:How the BPE tokenization algorithm used by large language... by cubefox on Thursday July 06, 2023 @10:14AM (#36616123)
> There are obviously more characters than the token vocabulary size of typical current models.

I'm pretty sure the set of tokens also contains all 256 Bytes to cover such cases.

Re:How the BPE tokenization algorithm used by large language... by weinzierl on Thursday July 06, 2023 @10:48AM (#36616819)
Not sure what you mean, but the transformer model can only predict one token at a time. The final output layer needs as many nodes (or neurons, if you will) as there are distinct in the token vocabulary. So a large token vocabulary is expensive and that's why GPT-3 and LLaMA have only about 50000 different tokens and use BPE to find a set of useful tokens. They still can express every possible English text because the token vocabulary contains the whole latin alphabet.

Unicode 15 has nearly 150000 characters and CJK languages have even more than that because of Han unification.

A model like GPT-3 can only output a very primitive version of Chinese. My question is how real Chinese models deal with this and specifically how tokenization works in that case.

Re:How the BPE tokenization algorithm used by large language... by space_fountain on Thursday July 06, 2023 @12:15PM (#36618282)
Yes, but the tokens are translated into bytes not characters. There are only 256 distinct bytes so GPT models can easily be trained to produce any character. Probably the problem will be how sensible or understandable the binary form of Chinese characters in Unicode are, but that will be a problem for the model, not the tokenizer
Re:How the BPE tokenization algorithm used by large language... by weinzierl on Thursday July 06, 2023 @12:59PM (#36619054)
Ok, I understand. That helped, thanks.

And also suddenly the B in BPE makes a lot of sense.

Re:How the BPE tokenization algorithm used by large language... by cubefox on Thursday July 06, 2023 @04:03PM (#36621985)
The way I think about it: A token can be one to many bytes long, so they can be longer or shorter than a single character.