Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Direct initialization of transformers using larger pretrained ones.
You can init deeper layers by copying the weights from previous layers. Not a big improvement in training time, but it does reduce the initial loss!
There are no comments on this comment yet.
You can init deeper layers by copying the weights from previous layers. Not a big improvement in training time, but it does reduce the initial loss!