Tokenization Is the Bottleneck You're Not Measuring Posted by mindsaspire on Monday May 25, 2026 @08:20PM from the ranvier.systems dept. https://ranvier.systems/2026/05/25/tokenization-is-the-bottleneck-youre-not-measuring.html ( 1 comment | on HN )
Github's BPE crate (https://crates.io/crates/bpe) advertises >10x speedup relative to HF, as do others.