News for hackers. Stuff that matters.
Re:Language Models Are Super Mario: Absorbing Abilities from... by m3kw9 on Saturday April 06, 2024 @11:36AM (#39953129)
Then why not start with a model that has all the weights zeroes out and start absorbing different models?
Re:Language Models Are Super Mario: Absorbing Abilities from... by l33tman on Saturday April 06, 2024 @12:15PM (#39953404)
It would be cool if there was like a threshold in training from scratch, where after this actually works for adding higher level knowledge. So you would start training it like usual to get it to absorb generic langauge skills and reasoning but save all the domain knowledge absorption for merging later
Re:Language Models Are Super Mario: Absorbing Abilities from... by skybrian on Saturday April 06, 2024 @01:02PM (#39953726)
The weights need to be connected to something interesting. Pre-training is how they get them all connected up, and fine-tuning is how they find which weights it would be useful to change.
Re:Language Models Are Super Mario: Absorbing Abilities from... by stoniejohnson on Saturday April 06, 2024 @02:29PM (#39954368)
Pretraining determines the weights (e.g. the connections), fine-tuning let's you change some subset of the weights (e.g. the final layers) with a smaller chunk of task-specific data.
Re:Language Models Are Super Mario: Absorbing Abilities from... by yumraj on Saturday April 06, 2024 @03:25PM (#39954737)
Isn’t x + 0 = x?

So what’s the benefit of starting with 0 when you can just start with x

I know it’s over simplistic above, but serious question, please ELI5..

Re:Language Models Are Super Mario: Absorbing Abilities from... by m3kw9 on Sunday April 07, 2024 @12:32AM (#39958203)
And then you apply a different models weights using the papers methods and then another