Re:How the BPE tokenization algorithm used by large language...
by space_fountain on Thursday July 06, 2023 @12:15PM
(#36618282)
Yes, but the tokens are translated into bytes not characters. There are only 256 distinct bytes so GPT models can easily be trained to produce any character. Probably the problem will be how sensible or understandable the binary form of Chinese characters in Unicode are, but that will be a problem for the model, not the tokenizer
Unicode 15 has nearly 150000 characters and CJK languages have even more than that because of Han unification.
A model like GPT-3 can only output a very primitive version of Chinese. My question is how real Chinese models deal with this and specifically how tokenization works in that case.