News for hackers. Stuff that matters.
Re:Norway's 2 petabytes of Huawei flash storage and LLM trai... by tecleandor on Tuesday May 26, 2026 @07:16AM (#48278133)
Tests I've done with NO and FI texts, for the same number of characters, with the GPT5 tokenizer I get around 2x the tokens than EN. With the older tokenizers it's more like 2x or even 3x.

There are no comments on this comment yet.