News for hackers. Stuff that matters.

A comment on Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken. Parent comment

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by 22c on Tuesday July 01, 2025 @08:05PM (#44439086)
There is at least some awareness already when it comes to the performance of the regex engine:

https://github.com/openai/tiktoken/blob/main/src/lib.rs#L95-...

[ Parent | on HN ]

There are no comments on this comment yet.