Re:Language Models Are Super Mario: Absorbing Abilities from...
by m3kw9 on Saturday April 06, 2024 @11:36AM
(#39953129)
Then why not start with a model that has all the weights zeroes out and start absorbing different models?
[ Parent
| on HN ]
Re:Language Models Are Super Mario: Absorbing Abilities from...
by l33tman on Saturday April 06, 2024 @12:15PM
(#39953404)
It would be cool if there was like a threshold in training from scratch, where after this actually works for adding higher level knowledge. So you would start training it like usual to get it to absorb generic langauge skills and reasoning but save all the domain knowledge absorption for merging later
Re:Language Models Are Super Mario: Absorbing Abilities from...
by skybrian on Saturday April 06, 2024 @01:02PM
(#39953726)
The weights need to be connected to something interesting. Pre-training is how they get them all connected up, and fine-tuning is how they find which weights it would be useful to change.
Re:Language Models Are Super Mario: Absorbing Abilities from...
by stoniejohnson on Saturday April 06, 2024 @02:29PM
(#39954368)
Pretraining determines the weights (e.g. the connections), fine-tuning let's you change some subset of the weights (e.g. the final layers) with a smaller chunk of task-specific data.