News for hackers. Stuff that matters.
Re:The Kimi K3 Moment by girvo on Saturday July 18, 2026 @04:55PM (#48962308)
Information is information. Why is some information considered different than others in your estimation?
Re:The Kimi K3 Moment by embedding-shape on Saturday July 18, 2026 @05:11PM (#48962457)
> Information is information

Not really, what the information actually is, matters a great deal. It's harder to get good results going from "nothing > model+weights" than "nothing + traces from known good sessions of other good model > model+weights", this is what the "distillation" part is referring to. If "information is information", you wouldn't even need to separate good from bad sessions while doing the training, which leads to somewhat obvious results if you don't.

Re:The Kimi K3 Moment by nonethewiser on Saturday July 18, 2026 @05:12PM (#48962467)
Can you be more specific? I have no idea what you are trying to say.

To succinctly restate my point, you cannot distill a model from information because the model is not contained within that information. You can distill a model from another model.

Re:The Kimi K3 Moment by margalabargala on Saturday July 18, 2026 @05:38PM (#48962698)
Their point is that "training" and "distillation" are essentially the same. The difference between the words is whether the source material is output from another model, vs being some original text.