News for hackers. Stuff that matters.
Re:Searching for DeepSeek's glitch tokens by brookst on Saturday January 25, 2025 @10:55PM (#42827571)
“Tokenizations are learned at the byte level” seems wrong. Tokens are integer representations of one or more characters, which themselves can be multiple bytes.

When you tokenize “ant” to 0x38 0xF9, it doesn’t matter if the original was three bytes of ascii or 0x00 0x00 0x00 0x61 0x00 0x00 0x00 0x6E 0x00 0x00 0x00 0x74

Re:Searching for DeepSeek's glitch tokens by mmoskal on Sunday January 26, 2025 @12:08AM (#42827851)
Tokens are in fact sequences of bytes not characters. For example, llama3 tokenizer (128k tokens) includes 1361 tokens that are invalid UTF8 (or rather they are partial UTF8).

Models will generally only produce valid UTF8 (that is when bytes of tokens are concatenated they are valid UTF8), unless really confused.

Re:Searching for DeepSeek's glitch tokens by petters on Sunday January 26, 2025 @02:45PM (#42833306)
They are, but they should not be