News for hackers. Stuff that matters.
Re:OpenAI Tokenizer by coffeeri on Wednesday April 05, 2023 @09:59AM (#35454326)
OpenAI seems to use Tiktoken [0]. It also covers GPT-4 token encoding.

[0] https://github.com/openai/tiktoken

Re:OpenAI Tokenizer by hombre_fatal on Wednesday April 05, 2023 @11:36AM (#35455941)
Of course, you can find BPE tokenizer impls in other languages for the same results as long as they use the "cl100k_base" data.

Typescript: https://github.com/danneu/telegram-chatgpt-bot/tree/master/t...

Re:OpenAI Tokenizer by lspears on Wednesday April 05, 2023 @12:50PM (#35457027)
Seems odd they don't reference this on the page. Instead they list:

"If you need a programmatic interface for tokenizing text, check out the transformers package for python or the gpt-3-encoder package for node.js."

with the links:

https://huggingface.co/docs/transformers/model_doc/gpt2#tran...

https://www.npmjs.com/package/gpt-3-encoder