Re:Anthropic Accuses Alibaba of 'Illicitly' Accessing AI Models
by sorenjan on Thursday June 25, 2026 @07:34AM
(#48671935)
Doesn't "real" distillation use the logits instead of the final tokens? I would classify this more like using a model to generate synthetic training data.
[ Parent
| on HN ]
Re:Anthropic Accuses Alibaba of 'Illicitly' Accessing AI Models
by 0xbadcafebee on Thursday June 25, 2026 @03:56PM
(#48678427)
Distillation is a category of techniques which generally speaking all extract knowledge from a target model to feed into a new model. Logit distillation requires access to the source model layers; final token distillation doesn't. The former is more effective, but the latter can be done with generation tokens alone.
This article explains the difference (and addresses "they're distillin' our models!!!"): https://dev.to/p0rt/how-model-distillation-actually-works-an...