Re:How the BPE tokenization algorithm used by large language...
by weinzierl on Thursday July 06, 2023 @11:16AM
(#36617308)
GPT uses BPE wit 50k or 100k token vocabulary from what I understand.
Given that a lot of space is taken by words and subwords this is not nearly enough for the CJK alphabet.
[ Parent
| on HN ]
Re:How the BPE tokenization algorithm used by large language...
by LoganDark on Thursday July 06, 2023 @12:34PM
(#36618601)
There are only 256 bytes. CJK characters can be produced by outputting these bytes in a certain order. LLMs are capable of outputting multiple tokens in order because even many words are multiple tokens each.