News for hackers. Stuff that matters.
Re:Why is GPT-3 15.77x more expensive for certain languages? by k8si on Monday April 10, 2023 @03:33PM (#35516387)
I don't know what you mean by compiler terms but basically, worse tokenizer = worse LM performance. This is because worse tokenizer means more tokens per sentence so it takes more FLOPs to train on each sentence, on average. So given a fixed training budget, English essentially gets more "learning per token" than other languages.

There are no comments on this comment yet.