News for hackers. Stuff that matters.
Re:How the BPE tokenization algorithm used by large language... by space_fountain on Thursday July 06, 2023 @12:12PM (#36618228)
It will happily translate simple phrases into Chinese and automated translation software can read it back. As other commenters are saying the tokenizer works on bytes so a single Chinese character will be made of multiple tokens just as some English words are made of multiple tokens. GPT doesn’t see characters it sees bytes. Probably it would do better with a encoding scheme for the characters that was designed around some sort of logic or consistency, but it can go far with just memorization

There are no comments on this comment yet.