Re:Better and Faster Large Language Models via Multi-Token P...
by jasonjmcghee on Wednesday May 01, 2024 @10:12AM
(#40223512)
what constitutes a token is super unintuitive.
my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters
In either case, I do consistently see spaces between characters included as part of the token following the space.
" paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is
my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters
In either case, I do consistently see spaces between characters included as part of the token following the space.
" paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is