News for hackers. Stuff that matters.

A comment on Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken.

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by janwilmake on Monday June 30, 2025 @10:18AM (#44423620)
You know what's also faster to roughly get the amount of tokens? string.length/5
[ Parent | on HN ]
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by _flux on Monday June 30, 2025 @01:26PM (#44425810)
It is not helpful in actual tokenization, though.
[ Parent | on HN ]