News for hackers. Stuff that matters.
Re:Deepseek R1-0528 by anonymoushn on Thursday May 29, 2025 @05:34AM (#44124407)
If labs provided the corpus and source code for training their tokenizers, it would be a lot easier to produce results about tokenizers. As it is, they provide neither, so it is impossible to compare different algorithms running on the same data if you also want to include the vocabs that are commonly used.

There are no comments on this comment yet.