Re:DeepMind’s New Language Model, Chinchilla
by danuker on Monday April 11, 2022 @11:09AM
(#30989872)
It might just be by chance: the initial weights of one model could have been lucky in some areas, and unlucky in others. There's no way to tell other than training again, which is a costly proposition.
[ Parent
| on HN ]
Re:DeepMind’s New Language Model, Chinchilla
by eigenvalue on Monday April 11, 2022 @05:33PM
(#30995393)
That seems pretty unlikely to me actually. As the models and training data get much bigger, I think the initial weights become less important (at least assuming your random weights have certain desirable statistical properties, which they do by construction usually).