Re:Direct initialization of transformers using larger pretra...
by peterfirefly on Friday December 22, 2023 @04:39PM
(#38738987)
Sounds like a good test.
We need something like this for "non-neural" things like transformers and normalization layers:
https://proceedings.mlr.press/v157/skorski21a/skorski21a.pdf
Re:Direct initialization of transformers using larger pretra...
by ausbah on Friday December 22, 2023 @06:02PM
(#38739642)
a better distribution of weights would be knowledge transfer right? by having something like a stronger prior distribution?
also pretty sure this would work because I know some text-image generators has the image network priority trained then frozen during e2e training
Re:Direct initialization of transformers using larger pretra...
by peterfirefly on Friday December 22, 2023 @06:48PM
(#38740100)
Not necessarily. It turns out to really matter how the weights are initialized (randomly, but from which distribution?) for ordinary feed-forward neural nets that only uses standard neurons -- different activation functions have different optimal initialization distributions.
https://machinelearningmastery.com/weight-initialization-for...
A big breakthrough was Kaiming He's work on initialization for ReLU:
https://arxiv.org/abs/1502.01852
(Kaiming is the personal name, He is the family name. He Kaiming is the native Chinese order. Lots of citations use "Kaiming" by mistake.)
A good test would be to apply this technique, but then train on a totally different kind of data - for example take a text LLM, apply this technique, but then train on audio/music data and see if this technique reduces training time over random initialization.