News for hackers. Stuff that matters.
Re:Collapse of Self-Trained Language Models by gwern on Wednesday April 17, 2024 @04:03PM (#40069479)
> One of the tricks OpenAI uses when fine-tuning models is to use the unadjusted foundation model as a coherency model alongside the data or reward model to be fine-tuned on. Loss is calculated as the sum[0] of both the fine-tuning and coherency loss so that the training process is anchored to something.

You mean the K-L constraint?

Re:Collapse of Self-Trained Language Models by tempusalaria on Wednesday April 17, 2024 @04:36PM (#40069806)
In InstructGPT there is a direct pretraining gradient addition as well as KL penalty