Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Differential Transformer. Parent comment
SoftmaxA[n] - SoftmaxB[n] is exactly 0?
Even if 2 attention layers learn two different things, I would imagine the corresponding weights in each layer wouldn’t exactly cancel each other out.
There are no comments on this comment yet.
SoftmaxA[n] - SoftmaxB[n] is exactly 0?
Even if 2 attention layers learn two different things, I would imagine the corresponding weights in each layer wouldn’t exactly cancel each other out.