Re:Transformer Attention is off by one
by janalsncm on Monday July 24, 2023 @05:37PM
(#36854557)
Sure, emergent properties can arise as parameters increase. Everyone knows that. That’s a much less specific claim than to say that the benefit of modifying softmax can only arise as an emergent property after N parameters, and therefore the benefit can only be evaluated on models above a certain size. To my understanding the author of TFA isn’t suggesting the same issue as the one in your linked paper.
[ Parent
| on HN ]
Re:Transformer Attention is off by one
by WithinReason on Monday July 24, 2023 @06:08PM
(#36854931)
The second heading in the TFA is "It’s All About Outliers"