Re:Searching for DeepSeek's glitch tokens
by mmoskal on Sunday January 26, 2025 @12:08AM
(#42827851)
Tokens are in fact sequences of bytes not characters. For example, llama3 tokenizer (128k tokens) includes 1361 tokens that are invalid UTF8 (or rather they are partial UTF8).
Models will generally only produce valid UTF8 (that is when bytes of tokens are concatenated they are valid UTF8), unless really confused.
When you tokenize “ant” to 0x38 0xF9, it doesn’t matter if the original was three bytes of ascii or 0x00 0x00 0x00 0x61 0x00 0x00 0x00 0x6E 0x00 0x00 0x00 0x74