Re:Searching for DeepSeek's glitch tokens
by pama on Sunday January 26, 2025 @01:48AM
(#42828247)
Right. I stand corrected and it makes sense. I may have misunderstood the OP. It would not make sense to encode each unicode point as a token and then start training the subword tokenizer on top unless we go to the vocabularies of many millions.
[ Parent
| on HN ]
There are no comments on this comment yet.