Direct initialization of transformers using larger pretrained ones

Re:Direct initialization of transformers using larger pretra... by ausbah on Friday December 22, 2023 @06:02PM (#38739642)

a better distribution of weights would be knowledge transfer right? by having something like a stronger prior distribution?

also pretty sure this would work because I know some text-image generators has the image network priority trained then frozen during e2e training

[ Parent | on HN ]

Re:Direct initialization of transformers using larger pretra... by peterfirefly on Friday December 22, 2023 @06:48PM (#38740100)

Not necessarily. It turns out to really matter how the weights are initialized (randomly, but from which distribution?) for ordinary feed-forward neural nets that only uses standard neurons -- different activation functions have different optimal initialization distributions.

https://machinelearningmastery.com/weight-initialization-for...

A big breakthrough was Kaiming He's work on initialization for ReLU:

https://arxiv.org/abs/1502.01852

(Kaiming is the personal name, He is the family name. He Kaiming is the native Chinese order. Lots of citations use "Kaiming" by mistake.)

[ Parent | on HN ]