News for hackers. Stuff that matters.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by cma on Monday May 01, 2023 @10:37AM (#35773364)
> The authors of Attention is all you need, the transformers paper, y

Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before:

https://arxiv.org/abs/2102.11174

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by mardifoufs on Monday May 01, 2023 @10:40AM (#35773405)
Schmidhuber claims to have invented a lot of things. It's almost a running gag at this point.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by uoaei on Monday May 01, 2023 @11:26AM (#35774140)
And yet somehow his claims always bear some truth. I understand the comments about boys crying wolf, but it's hard to ignore the facts on the ground.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by caycep on Monday May 01, 2023 @02:16PM (#35776879)
not having a PHD in ML, it's hard for me to evaluate his claims, but how valid are all the obscure papers that he brings up? Did someone actually invent backprop in 1930 in some random corner of the former Soviet Union? Or is it a case of "true but misses the point"?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by uoaei on Monday May 01, 2023 @02:47PM (#35777289)
Often it is indeed the latter, although it is interesting that sometimes despite that it gets at the core of our contemporary understanding of the concepts in question.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by telotortium on Monday May 01, 2023 @02:44PM (#35777263)
"Formal equivalence" means very little for engineering, to be frank - the implementation is the important thing. If I wanted to be snarky, I'd say that neural networks are "formally equivalent" to Fourier analysis, which is 200 years old. I see that the paper proposes an implementation of linearized attention as well, which many others have done, but none of which seem to have caught on (although FlashAttention at least makes attention O(n) in memory, if not computation).