Re:Norway's 2 petabytes of Huawei flash storage and LLM trai...
by tecleandor on Tuesday May 26, 2026 @07:16AM
(#48278133)
Tests I've done with NO and FI texts, for the same number of characters, with the GPT5 tokenizer I get around 2x the tokens than EN. With the older tokenizers it's more like 2x or even 3x.
English performs the best because there is more data in English and high quality sources are either only in English or there is a good translation in English.
[1]: https://platform.openai.com/tokenizer