Re:Anthropic announces proof of distillation at scale by Min...
by roborovskis on Monday February 23, 2026 @02:36PM
(#47127586)
What would you define as 'distillation' versus 'learning'? How do you know that what a LLM is doing is 'distillation' vs a process closer to a human reading a book?
From my perspective, pretraining is pretty clearly not 'distilling', as the goal is not to replicate the pretraining data but to generalize. But what these companies are doing is clearly 'distilling' in that they want their models to exactly emulate Claude's behavior.
Re:Anthropic announces proof of distillation at scale by Min...
by armcat on Monday February 23, 2026 @02:53PM
(#47127817)
That's a soft distinction (distilling vs learning). If I read a chapter in a text book I am distilling the knowledge from that chapter into my own latent space - one would hope I learn something. Flipping it the other way, you could say that model from Lab Y is ALSO learning the model from Lab X. Not just "distilling". Hence my original comment - how deep does this go?
Re:Anthropic announces proof of distillation at scale by Min...
by EnPissant on Monday February 23, 2026 @03:08PM
(#47128030)
And yet nearly every machine learning engineer would disagree with you, which is a given away that your argument is rooted in ideology.
Re:Anthropic announces proof of distillation at scale by Min...
by armcat on Monday February 23, 2026 @03:21PM
(#47128206)
> And yet nearly every machine learning engineer would disagree with you, which is a given away that your argument is rooted in ideology.
That's a bold statement! Of course I know the difference, in one case you are learning from correct/wrong answers, and in the other from a probability distribution. But in both cases you are using some X to move the weights. We can get down and gritty on KL divergence vs cross-entropy, but the whole topic is about "theft", which is perhaps in the eye of the beholder.
Then another lab comes, and "steals" from you - that beautiful, refined dataoil - by distilling your weights using inferior equipment but with a toolbox of ingenuity and low-level hacking tricks. They reach 90% of your performance at 20x cost reduction.
What happens when another lab distills from the distilled lab?
Who is the thief? How far will the Alice go?