News for hackers. Stuff that matters.

A comment on Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken.

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by pamelafox on Monday June 30, 2025 @10:41AM (#44423953)
Just curious whether it's possible to push any of your performance improvements to tiktoken itself?
[ Parent | on HN ]
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by matthewolfe on Monday June 30, 2025 @10:48AM (#44424042)
I probably will. Was hesitant initially, because adding PCRE2 as a dependency might cause issues to existing projects. I believe this was discussed briefly in a closed PR with other performance improvements.
[ Parent | on HN ]