News for hackers. Stuff that matters.

A comment on tiktoken: OpenAI’s Tokenizer. Parent comment

Re:tiktoken: OpenAI’s Tokenizer by jraph on Friday December 16, 2022 @04:37AM (#34012631)
Of course, it's because you are not tokenizing it right. It's tik-token.

(Could we say it is kind of a garden-path word?)

[ Parent | on HN ]

There are no comments on this comment yet.