Re:Norway's 2 petabytes of Huawei flash storage and LLM trai...
by YetAnotherNick on Tuesday May 26, 2026 @02:13AM
(#48275679)
Did you even try to verify your claims. I tested it on few translations on wikipedia articles using [1] and it takes 15-20% more tokens for Norwegian.
English performs the best because there is more data in English and high quality sources are either only in English or there is a good translation in English.
Re:Norway's 2 petabytes of Huawei flash storage and LLM trai...
by tecleandor on Tuesday May 26, 2026 @07:16AM
(#48278133)
Tests I've done with NO and FI texts, for the same number of characters, with the GPT5 tokenizer I get around 2x the tokens than EN. With the older tokenizers it's more like 2x or even 3x.
There are a lot of efforts to create LLMs for dying languages and others that use cross cultural models to boost, but if your language is well literate, there’s a good reason to build a heritage LLM specific to your language and culture. Expecting OpenAI or Anthropic to prioritize your language over their target audience when a tradeoff is to be made is absurd.