Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on tiktoken: OpenAI’s Tokenizer. Parent comment
This particular tokenizer is very interesting given that it tries to be best of both worlds (word-level tokenizer and character-level).
There are no comments on this comment yet.
This particular tokenizer is very interesting given that it tries to be best of both worlds (word-level tokenizer and character-level).