News for hackers. Stuff that matters.
Re:Simplifying Transformer Blocks by jksk61 on Tuesday November 28, 2023 @05:32AM (#38444352)
also removing skip connections leads to a rougher loss landscape, hence it should be harder to find the optimal weights.
Re:Simplifying Transformer Blocks by sdenton4 on Tuesday November 28, 2023 @06:09AM (#38444565)
Yes there's very good theoretical reasons for skip connections. If your initial matrix M is noise centered at 0, then 1+M is a noisy identity operation, while 0+M is a noisy deletion... It's better to do nothing if you don't know what to do, and avoid destroying information.

I appreciate the sibling comment perspective that memory pressure is a problem, but that can be mediated by using fewer/longer skip connections across blocks of layers.