Re:Better and Faster Large Language Models via Multi-Token P...
by ralusek on Wednesday May 01, 2024 @08:31AM
(#40222242)
Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words.
If you want to just see the tokens for yourself, though, just enter some text here:
Re:Better and Faster Large Language Models via Multi-Token P...
by sanxiyn on Wednesday May 01, 2024 @09:07AM
(#40222632)
No, it isn't, at least on OpenAI tokenizer.
Re:Better and Faster Large Language Models via Multi-Token P...
by jasonjmcghee on Wednesday May 01, 2024 @10:12AM
(#40223512)
what constitutes a token is super unintuitive.
my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters
In either case, I do consistently see spaces between characters included as part of the token following the space.
" paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is
Re:Better and Faster Large Language Models via Multi-Token P...
by jasonjmcghee on Wednesday May 01, 2024 @10:18AM
(#40223591)
New winner " communication" at 14
I'm a total amateur in this field though.