News for hackers. Stuff that matters.
Re:Magistral — the first reasoning model by Mistral AI by Onavo on Tuesday June 10, 2025 @11:26AM (#44238026)
> Removed KL Divergence

Wait, how are they computing the loss?

Re:Magistral — the first reasoning model by Mistral AI by danielhanchen on Tuesday June 10, 2025 @11:32AM (#44238091)
Oh it's the KL term sorry - beta * KL ie they set beta to 0.

The goal of it was to "force" the model not to stray to far away from the original checkpoint, but it can hinder the model from learning new things

Re:Magistral — the first reasoning model by Mistral AI by mjburgess on Tuesday June 10, 2025 @11:44AM (#44238218)
It's just a penalty term that they delete
Re:Magistral — the first reasoning model by Mistral AI by trc001 on Tuesday June 10, 2025 @06:37PM (#44242276)
It's become trendy to delete it. I say trendy because many papers delete it without offering any proof that it is meaningless