News for hackers. Stuff that matters.
Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @02:40PM (#36852495)
* * *
Re:Transformer Attention is off by one by janalsncm on Monday July 24, 2023 @02:50PM (#36852627)
That isn’t true at all. Train a smaller model on a smaller dataset. You can even train on your laptop. It’s definitely feasible. This is just a proof of concept, it doesn’t need to beat state of the art.
Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @03:18PM (#36852975)
Maybe I edited my comment too late.
Re:Transformer Attention is off by one by janalsncm on Monday July 24, 2023 @03:44PM (#36853251)
> I believe the outlier problem that this solves only appears for very large models.

Any reason to believe this? The author never mentioned it, and I can’t think of any other a priori reason why it should be true.

Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @04:41PM (#36853908)
* * *
Re:Transformer Attention is off by one by janalsncm on Monday July 24, 2023 @05:37PM (#36854557)
Sure, emergent properties can arise as parameters increase. Everyone knows that. That’s a much less specific claim than to say that the benefit of modifying softmax can only arise as an emergent property after N parameters, and therefore the benefit can only be evaluated on models above a certain size. To my understanding the author of TFA isn’t suggesting the same issue as the one in your linked paper.
Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @06:08PM (#36854931)
The second heading in the TFA is "It’s All About Outliers"
Re:Transformer Attention is off by one by PoignardAzur on Monday July 24, 2023 @06:18PM (#36855021)
6.7B isn't "needs a datacenter" scale.
Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @06:28PM (#36855130)
It's in the million dollar range. XLnet which is a 1.3B model cost $245,000 to train for example.