News for hackers. Stuff that matters.
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by pama on Monday June 30, 2025 @10:08AM (#44423489)
Cool. Would it be possible to eliminate that little vocab format conversion requirement for the vocab I see in the test against tiktoken? It would be nice to have a fully compatible drop in replacement without having to think about details. It also would be nice to have examples that work the other way around: initialize tiktoken as you normally would, including any specialized extension of standard tokenizers, and then use that initialized tokenizer to initialize a new tokendagger and test identity of results.
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by matthewolfe on Monday June 30, 2025 @12:25PM (#44425102)
Ah good catch. Updating this right now.
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by matthewolfe on Monday June 30, 2025 @12:57PM (#44425489)
Alright, 0.1.1 should now be a true drop-in replacement. I'll write up some examples soon.