Re:Claude Code's "extended thinking" is a summary- not authe...
by wren6991 on Monday June 22, 2026 @01:55PM
(#48633552)
> Proper distillation requires access to the logits
Why do you need logits? Can't you just train on cross-entropy loss of the model against the hard decision, like you do in regular pretraining?
There are definitely current-gen open-weight models (Step 3.7 Flash is one) that refer to themselves as an OpenAI model in CoT, but not in the final response.
Re:Claude Code's "extended thinking" is a summary- not authe...
by CamperBob2 on Monday June 22, 2026 @02:25PM
(#48633958)
How do I get that loss, though, without the softmax inputs?
It only makes sense that the same mechanism comes into play in strictly-verbal contexts.
Also, this is why "distillation attacks" are largely bullshit that Anthropic spreads for political purposes. Proper distillation requires access to the logits.