News for hackers. Stuff that matters.

A comment on Introducing ChatGPT and Whisper APIs. Parent comment

Re:Introducing ChatGPT and Whisper APIs by triyambakam on Wednesday March 01, 2023 @02:56PM (#34987442)
Can you explain what tokens are in this context?

Edit: and better yet, is there a good resource for learning the vernacular in general? Should I just read something like "Dive into Deep Learning"?

[ Parent | on HN ]
Re:Introducing ChatGPT and Whisper APIs by jncraton on Wednesday March 01, 2023 @03:32PM (#34987887)
If an example would be helpful, OpenAI's tokenizer is publicly usable on their website:

https://platform.openai.com/tokenizer

You can drop sample text in there and visually see how it is split into tokens. The GPT2/3 tokenizer uses about 50k unique tokens that were learned to be an efficient representation of the training data.

[ Parent | on HN ]
Re:Introducing ChatGPT and Whisper APIs by dmw_ng on Wednesday March 01, 2023 @03:07PM (#34987560)
https://en.wikipedia.org/wiki/Byte_pair_encoding

In short, common words cost one token, rarer words cost a few tokens

[ Parent | on HN ]
Re:Introducing ChatGPT and Whisper APIs by aitball on Wednesday March 01, 2023 @03:14PM (#34987664)
no the language model decides what a token is
[ Parent | on HN ]