News for hackers. Stuff that matters.

A comment on Cognition launches new SWE-2 model, Rivaling Fable 5.1 and GPT-Astra. Parent comment

Re:Cognition launches new SWE-2 model, Rivaling Fable 5.1 an... by FergusArgyll on Thursday September 10, 2026 @02:29PM (#49648258)
Distilling you don't have the actual model weights of the teacher. All you have are the teachers answers to a lot of questions. You then teach your own smaller model to answer more similarly to the big teacher model.

Fine tuning you have the actual model weights of the original model, you then train that model to answer in a different (or better) way.

[ Parent | on HN ]
Re:Cognition launches new SWE-2 model, Rivaling Fable 5.1 an... by Bolwin on Thursday September 10, 2026 @10:23PM (#49652838)
Distillation requires you to have the actual logits of each token from the teacher model, which in practice means having the model itself.

What you're describing is just synthetic data.

Note Anthropic misused the term in their post about Chinese model distillation, deliberately I assume.

[ Parent | on HN ]
Re:Cognition launches new SWE-2 model, Rivaling Fable 5.1 an... by FergusArgyll on Thursday September 10, 2026 @11:24PM (#49653190)
There are multiple kinds of distillation

https://arxiv.org/abs/2106.03310

https://arxiv.org/abs/2207.12106

[ Parent | on HN ]