News for hackers. Stuff that matters.

A comment on tiktoken: OpenAI’s Tokenizer. Parent comment

Re:tiktoken: OpenAI’s Tokenizer by DanSmooth on Friday December 16, 2022 @04:51AM (#34012717)
Same. A shame "Token McTokenface" seems to be already in use. What about "Sentence Splitter Upper"?

More serious "qiktoken" would at least hint at the use for the package.

[ Parent | on HN ]

There are no comments on this comment yet.