Re:Apple: Embarrassingly Simple Self-Distillation Improves C...
by grumbelbart on Sunday April 05, 2026 @06:34AM
(#47648010)
Is this some kind of calibration then? I'd expect that the probabilities automatically adjust during training, such that in "lock" mode, for example, syntax-breaking tokens have a very low probability and would not be picked even wich higher temperature.
Re:Apple: Embarrassingly Simple Self-Distillation Improves C...
by tatrions on Sunday April 05, 2026 @02:54PM
(#47652636)
Token-level entropy is a pretty clean proxy for detecting fork vs lock positions at inference time. Low entropy = lock (decode greedily), high entropy = fork (sample with more temperature). Speculative decoding already exploits something similar where the small draft model handles the predictable tokens and the big model kicks in at the uncertain ones. Combining that with this paper's fork/lock framing could get you adaptive temperature basically for free during inference.
Maybe I'll try that myself, because it feels like it could be a great source of improvements. It would be really useful to see adaptive per-token sampling as an additional decode-only baseline.