Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by chrismustcode on Monday June 30, 2025 @09:06AM (#44422818)

There’s something beautiful about creating a drop in replacement for something that improves performance substantially.

ScyllaDB comes to mind

[ Parent | on HN ]

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by matthewolfe on Monday June 30, 2025 @09:08AM (#44422833)

Agreed. I figured nobody would use it otherwise.

[ Parent | on HN ]

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by parhamn on Monday June 30, 2025 @09:11AM (#44422860)

Put it in there readme & description. It's a big selling point.

[ Parent | on HN ]

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by matthewolfe on Monday June 30, 2025 @09:16AM (#44422914)

Thanks, I clarified it.

[ Parent | on HN ]

Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by pvg on Monday June 30, 2025 @09:14AM (#44422898)

To be fair, many people have token stabbing needs.

[ Parent | on HN ]