News for hackers. Stuff that matters.
Re:What Are Transformer Models and How Do They Work? by montebicyclelo on Saturday April 15, 2023 @11:49AM (#35581721)
OpenAI have made their tokenizers public [1].

As someone has pointed out, with BPE you specify the vocab size, not the token size. It's a relatively simple algo, this Huggingface course does a nice job of explaining it [2]. Plus the original paper has a very readable Python example [3].

[1] https://github.com/openai/tiktoken

[2] https://huggingface.co/course/chapter6/5?fw=pt

[3] https://arxiv.org/abs/1508.07909

There are no comments on this comment yet.