Re:Why is GPT-3 15.77x more expensive for certain languages?
by kevingadd on Monday April 10, 2023 @02:42PM
(#35515792)
The behavior on Chinese is what makes me believe it's tokenizing on something like UTF-8 (hopefully normalized). I'm not sure how else you would get that behavior.
Tokens for non-english languages that are groups of characters just suggests that common groups of 2-3 characters from the training set became tokens, which feels unsurprising. The fallback behavior would be 1 utf8 byte = 1 token.
Re:Why is GPT-3 15.77x more expensive for certain languages?
by Vecr on Monday April 10, 2023 @03:06PM
(#35516069)
That might not be true. OpenAI do set a limit of the total number of tokens, and since I'm pretty sure they trained the model and the tokenizer on mostly English text, I assume there's a somewhat proportional bias toward English based on the input dataset to those models.
But when it comes to Chinese...something weird is going on.