News for hackers. Stuff that matters.
Re:Tokenization Is the Bottleneck You're Not Measuring by olaird25 on Tuesday May 26, 2026 @01:31PM (#48282913)
Have you tried just using a faster tokenization library?

Github's BPE crate (https://crates.io/crates/bpe) advertises >10x speedup relative to HF, as do others.