News for hackers. Stuff that matters.
Re:Transformer Attention is off by one by chessgecko on Monday July 24, 2023 @02:24PM (#36852258)
I think maybe its because he didn't have experimental results that show that it worked. Not a knock against the author, there are just so many things that seem like good ideas that don't end up working well in practice, a paper like this without results is hard to value.
Re:Transformer Attention is off by one by mlsu on Monday July 24, 2023 @02:40PM (#36852482)
Yes, definitely. If he tried to have it published, the lack of experimental results would definitely be a glaring error.

But this is still scientific communication. It's really nice that it's legible!

> Even though softmax1 is facially quite boring, I’m 99.44% sure that it will resolve the outlier feedback loop that’s making quantization the subject of cascades of research. If you want to run some experiments and prove me right, DM me on Twitter and we’ll get a paper going.

I'm guessing that in the stodgy world of science, a communication like this might happen over lunch at a conference, limited to a small clique of researchers who are zealously guarding their next paper. Who could blame them, publish or perish!

But someone will probably test this theory out (after my read, it will probably happen in llama.cpp with preliminary results on GPT-2 by next week) and achieve results, and it will happen quickly and legibly to the outside world, because this was published openly and without all of the pretension that formal science (tm) has. If it works, it works. Stuff like this is the soul of the internet. Sharing knowledge and making it legible for all.

Re:Transformer Attention is off by one by light_hue_1 on Monday July 24, 2023 @07:39PM (#36855746)
There's a perfectly good venue for this communication: a workshop.

Workshop submissions often don't need evidence. They just need a small kernel to spur discussion.

Without experiments, there is no hope of publishing this in anything more than a workshop. Nor should there be.

Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @02:40PM (#36852495)
* * *
Re:Transformer Attention is off by one by janalsncm on Monday July 24, 2023 @02:50PM (#36852627)
That isn’t true at all. Train a smaller model on a smaller dataset. You can even train on your laptop. It’s definitely feasible. This is just a proof of concept, it doesn’t need to beat state of the art.
Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @03:18PM (#36852975)
Maybe I edited my comment too late.
Re:Transformer Attention is off by one by janalsncm on Monday July 24, 2023 @03:44PM (#36853251)
> I believe the outlier problem that this solves only appears for very large models.

Any reason to believe this? The author never mentioned it, and I can’t think of any other a priori reason why it should be true.

Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @04:41PM (#36853908)
* * *
Re:Transformer Attention is off by one by janalsncm on Monday July 24, 2023 @05:37PM (#36854557)
Sure, emergent properties can arise as parameters increase. Everyone knows that. That’s a much less specific claim than to say that the benefit of modifying softmax can only arise as an emergent property after N parameters, and therefore the benefit can only be evaluated on models above a certain size. To my understanding the author of TFA isn’t suggesting the same issue as the one in your linked paper.
Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @06:08PM (#36854931)
The second heading in the TFA is "It’s All About Outliers"
Re:Transformer Attention is off by one by PoignardAzur on Monday July 24, 2023 @06:18PM (#36855021)
6.7B isn't "needs a datacenter" scale.
Re:Transformer Attention is off by one by WithinReason on Monday July 24, 2023 @06:28PM (#36855130)
It's in the million dollar range. XLnet which is a 1.3B model cost $245,000 to train for example.