Re:Better and Faster Large Language Models via Multi-Token P...
by ralusek on Wednesday May 01, 2024 @08:31AM
(#40222242)
Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words.
If you want to just see the tokens for yourself, though, just enter some text here:
If you want to just see the tokens for yourself, though, just enter some text here:
https://platform.openai.com/tokenizer