Re:Xkcd 1425 (Tasks) turns ten years old today
by conesus on Thursday September 26, 2024 @01:15PM
(#41660850)
GPT-4o uses BPE (byte pair encoding). They released `tiktoken` which allows you to count tokens in strings in python.
[ Parent
| on HN ]
There are no comments on this comment yet.