News for hackers. Stuff that matters.
Re:Why is GPT-3 15.77x more expensive for certain languages? by tlrobinson on Monday April 10, 2023 @02:33PM (#35515666)
I think yes, but more precisely the tokens were chosen to optimize for a dataset that's biased to English content.

I am curious how the token set affects quality of responses, ignoring the factors related to token count mentioned in the post (cost, prompt expressivity, latency, etc)

Is it always better for the token set to be "native" to the majority of the training dataset and prompts/completions, or is it possible there's some "intermediate representation" (in compiler terms) that would be better?

Re:Why is GPT-3 15.77x more expensive for certain languages? by k8si on Monday April 10, 2023 @03:33PM (#35516387)
I don't know what you mean by compiler terms but basically, worse tokenizer = worse LM performance. This is because worse tokenizer means more tokens per sentence so it takes more FLOPs to train on each sentence, on average. So given a fixed training budget, English essentially gets more "learning per token" than other languages.