News for hackers. Stuff that matters.

A comment on Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken. Parent comment

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by notatallshaw on Monday June 30, 2025 @05:15PM (#44427963)
It looks like TikToken is written in Rust (https://github.com/openai/tiktoken/tree/main/src), are the gains here actually from porting to C++?
[ Parent | on HN ]
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by fhub on Monday June 30, 2025 @11:11PM (#44430216)
From the post

Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.

[ Parent | on HN ]