News for hackers. Stuff that matters.

A comment on Direct initialization of transformers using larger pretrained ones.

Re:Direct initialization of transformers using larger pretra... by DarmokJalad1701 on Friday December 22, 2023 @04:12PM (#38738724)
From Page 7:

> In the case of GPT-2 training, random initialization requires 64×10^9 tokens to reach a perplexity of 12, while weight subcloning accomplishes this in just 64×10^9 tokens, again demonstrating a 4× training speedup

... what?

[ Parent | on HN ]
Re:Direct initialization of transformers using larger pretra... by mirekrusin on Friday December 22, 2023 @04:47PM (#38739054)
Looks like typo, from chart it seems to be around 20x10^9 for weight subcloning (doesn't look like 16x10^9 <<which would be 4x>> either).
[ Parent | on HN ]