News for hackers. Stuff that matters.

A comment on Differential Transformer. Parent comment

Re:Differential Transformer by schopra909 on Tuesday October 08, 2024 @11:19AM (#41778315)
Quite the opposite — if you have a long sequence only a smattering of the words will influence the meaning of the current word. Everything else is “noise”.

Attention is really good at finding this smattering of words (ie assign most weight there). But it struggles to put exactly 0 on the other words.

[ Parent | on HN ]
Re:Differential Transformer by absoflutely on Tuesday October 08, 2024 @12:26PM (#41779079)
why say lot word when few word do
[ Parent | on HN ]
Re:Differential Transformer by dartos on Tuesday October 08, 2024 @01:19PM (#41779574)
Few word no do tho
[ Parent | on HN ]
Re:Differential Transformer by kridsdale3 on Tuesday October 08, 2024 @03:38PM (#41781012)
U+1FAE5
[ Parent | on HN ]
Re:Differential Transformer by 1024core on Tuesday October 08, 2024 @02:34PM (#41780356)
Phew!
[ Parent | on HN ]
Re:Differential Transformer by dartos on Tuesday October 08, 2024 @01:21PM (#41779598)
I mean wouldn’t it be unlikely that

SoftmaxA[n] - SoftmaxB[n] is exactly 0?

Even if 2 attention layers learn two different things, I would imagine the corresponding weights in each layer wouldn’t exactly cancel each other out.

[ Parent | on HN ]