News for hackers. Stuff that matters.
Re:How the BPE tokenization algorithm used by large language... by LoganDark on Thursday July 06, 2023 @12:34PM (#36618601)
There are only 256 bytes. CJK characters can be produced by outputting these bytes in a certain order. LLMs are capable of outputting multiple tokens in order because even many words are multiple tokens each.

There are no comments on this comment yet.