Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T...
by pama on Monday June 30, 2025 @10:08AM
(#44423489)
Cool. Would it be possible to eliminate that little vocab format conversion requirement for the vocab I see in the test against tiktoken? It would be nice to have a fully compatible drop in replacement without having to think about details. It also would be nice to have examples that work the other way around: initialize tiktoken as you normally would, including any specialized extension of standard tokenizers, and then use that initialized tokenizer to initialize a new tokendagger and test identity of results.
[ Parent
| on HN ]
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T...
by matthewolfe on Monday June 30, 2025 @12:25PM
(#44425102)
Ah good catch. Updating this right now.
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T...
by matthewolfe on Monday June 30, 2025 @12:57PM
(#44425489)
Alright, 0.1.1 should now be a true drop-in replacement. I'll write up some examples soon.