News for hackers. Stuff that matters.
Re:Norway's 2 petabytes of Huawei flash storage and LLM trai... by YetAnotherNick on Tuesday May 26, 2026 @02:13AM (#48275679)
Did you even try to verify your claims. I tested it on few translations on wikipedia articles using [1] and it takes 15-20% more tokens for Norwegian.

English performs the best because there is more data in English and high quality sources are either only in English or there is a good translation in English.

[1]: https://platform.openai.com/tokenizer

Re:Norway's 2 petabytes of Huawei flash storage and LLM trai... by tecleandor on Tuesday May 26, 2026 @07:16AM (#48278133)
Tests I've done with NO and FI texts, for the same number of characters, with the GPT5 tokenizer I get around 2x the tokens than EN. With the older tokenizers it's more like 2x or even 3x.
Re:Norway's 2 petabytes of Huawei flash storage and LLM trai... by numpad0 on Tuesday May 26, 2026 @07:44AM (#48278385)
Tokenizer efficiency varying by languages, by as much as up to 15x, is very well known and established

  https://www.google.com/search?q=tokenizer+efficiency+by+language