News for hackers. Stuff that matters.
Re:Searching for DeepSeek's glitch tokens by pama on Sunday January 26, 2025 @01:48AM (#42828247)
Right. I stand corrected and it makes sense. I may have misunderstood the OP. It would not make sense to encode each unicode point as a token and then start training the subword tokenizer on top unless we go to the vocabularies of many millions.

There are no comments on this comment yet.