News for hackers. Stuff that matters.
Re:tiktoken: OpenAI’s Tokenizer by mnks on Friday December 16, 2022 @04:51AM (#34012716)
Have a look at https://beta.openai.com/tokenizer which uses javascript reimplementation of the GPT-2 / GPT-3 BPE tokenizer. In this case it's [31373, 995].

There are no comments on this comment yet.