News for hackers. Stuff that matters.
Re:Better and Faster Large Language Models via Multi-Token P... by ralusek on Wednesday May 01, 2024 @08:31AM (#40222242)
Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words.

If you want to just see the tokens for yourself, though, just enter some text here:

https://platform.openai.com/tokenizer

There are no comments on this comment yet.