Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by kevmo314 on Monday June 30, 2025 @10:56AM (#44424134)

Nice work! I tried something similar a while back ago: https://github.com/kevmo314/tokie

The takeaway I also found was that the running cost was really dominated by pretokenization (the regex). It's cool to see that you found a faster way to run the regex, but have you tried comparing the performance of just swapping out the regex engine and leaving the actual BPE to tiktoken? I wonder if that is upstreamable?

[ Parent | on HN ]

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by matthewolfe on Monday June 30, 2025 @11:12AM (#44424322)

Cool!

I've reached out to the guy who maintains Tiktoken to talk about this.

[ Parent | on HN ]

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by 22c on Tuesday July 01, 2025 @08:05PM (#44439086)

There is at least some awareness already when it comes to the performance of the regex engine:

https://github.com/openai/tiktoken/blob/main/src/lib.rs#L95-...

[ Parent | on HN ]