Re:Norway's 2 petabytes of Huawei flash storage and LLM trai...
by tecleandor on Tuesday May 26, 2026 @07:16AM
(#48278133)
Tests I've done with NO and FI texts, for the same number of characters, with the GPT5 tokenizer I get around 2x the tokens than EN. With the older tokenizers it's more like 2x or even 3x.
[ Parent
| on HN ]
There are no comments on this comment yet.