News for hackers. Stuff that matters.
Re:Norway's 2 petabytes of Huawei flash storage and LLM trai... by numpad0 on Tuesday May 26, 2026 @07:44AM (#48278385)
Tokenizer efficiency varying by languages, by as much as up to 15x, is very well known and established

  https://www.google.com/search?q=tokenizer+efficiency+by+language

There are no comments on this comment yet.