News for hackers. Stuff that matters.
Re:Why is GPT-3 15.77x more expensive for certain languages? by kevingadd on Monday April 10, 2023 @02:42PM (#35515792)
The behavior on Chinese is what makes me believe it's tokenizing on something like UTF-8 (hopefully normalized). I'm not sure how else you would get that behavior.

Tokens for non-english languages that are groups of characters just suggests that common groups of 2-3 characters from the training set became tokens, which feels unsurprising. The fallback behavior would be 1 utf8 byte = 1 token.

Re:Why is GPT-3 15.77x more expensive for certain languages? by Vecr on Monday April 10, 2023 @03:06PM (#35516069)
That might not be true. OpenAI do set a limit of the total number of tokens, and since I'm pretty sure they trained the model and the tokenizer on mostly English text, I assume there's a somewhat proportional bias toward English based on the input dataset to those models.