News for hackers. Stuff that matters.
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by stavros on Monday June 30, 2025 @11:47AM (#44424691)
Yeah, if it's not right, it doesn't work.
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by DSingularity on Monday June 30, 2025 @12:06PM (#44424906)
Not true. Things can work with hacks. Your standards might consider it unacceptable to have hacks. So you can have a “make it right” stage.
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by darknoon on Monday June 30, 2025 @11:54AM (#44424765)
In ML, often it does work to a degree even if it's not 100% correct. So getting it working at all is all about hacking b/c most ideas are bad and don't work. Then you'll find wins by incrementally correcting issues with the math / data / floating point precision / etc.
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by gabrielhidasy on Monday June 30, 2025 @01:24PM (#44425794)
Depends on your definition of "right" and "work". It could be a big ball of mud that always returns exactly the required response (so it 'works'), but be hellish hard change and very picky about dependencies and environment (so it's not 'right').
Re:Show HN: TokenDagger – A tokenizer faster than OpenAI's T... by stavros on Monday June 30, 2025 @01:29PM (#44425846)
Nope, it's right, but it's not pretty.