Re:Differential Transformer
by kelseyfrog on Tuesday October 08, 2024 @11:39AM
(#41778568)
The values between the groups are also going to diverge during training due to the structure of the DiffAttn equation.
The analogy I can think of is when you're paying attention to a variety of things and you actively avoid concentrating on something because it will distract you. You don't give it zero attention, you give it negative attention.
The analogy I can think of is when you're paying attention to a variety of things and you actively avoid concentrating on something because it will distract you. You don't give it zero attention, you give it negative attention.