Re:GigaToken: ~1000x faster Language model tokenization
by marcelroed on Wednesday July 22, 2026 @02:58PM
(#49011719)
Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.
[ Parent
| on HN ]
Re:GigaToken: ~1000x faster Language model tokenization
by robotresearcher on Wednesday July 22, 2026 @05:15PM
(#49013534)
> a modest ~200-300x speedup with compatibility mode
marcelroed is modest, this speedup is not. Good work.