Re:How the BPE tokenization algorithm used by large language...
by cubefox on Thursday July 06, 2023 @04:03PM
(#36621985)
The way I think about it: A token can be one to many bytes long, so they can be longer or shorter than a single character.
[ Parent
| on HN ]
There are no comments on this comment yet.