News for hackers. Stuff that matters.

A comment on Ask HN: What Are You Working On? (June 2025).

Re:Ask HN: What Are You Working On? (June 2025) by matthewolfe on Sunday June 29, 2025 @06:35PM (#44417208)
I'm working on TokenDagger [0] a high performance implementation of OpenAI's Tiktoken. My benchmarks are showing 2-3x higher throughput, as well as ~4x faster tokenization for code samples on a single thread.

[0] https://github.com/M4THYOU/TokenDagger

[ Parent | on HN ]

There are no comments on this comment yet.