News for hackers. Stuff that matters.
Re:Searching for DeepSeek's glitch tokens by mmoskal on Sunday January 26, 2025 @12:08AM (#42827851)
Tokens are in fact sequences of bytes not characters. For example, llama3 tokenizer (128k tokens) includes 1361 tokens that are invalid UTF8 (or rather they are partial UTF8).

Models will generally only produce valid UTF8 (that is when bytes of tokens are concatenated they are valid UTF8), unless really confused.

Re:Searching for DeepSeek's glitch tokens by petters on Sunday January 26, 2025 @02:45PM (#42833306)
They are, but they should not be