News for hackers. Stuff that matters.

A comment on tiktoken: OpenAI’s Tokenizer. Parent comment

Re:tiktoken: OpenAI’s Tokenizer by jeroenvlek on Friday December 16, 2022 @03:19AM (#34012148)
Huggingface is a lot more than just a tokenizer.
[ Parent | on HN ]

There are no comments on this comment yet.