Re:Geoffrey Hinton leaves Google and can now speak freely ab...
by KRAKRISMOTT on Monday May 01, 2023 @10:21AM
(#35773144)
If you need to build an airplane, would you rather consult Newton, the Wright brothers, or a modern aerospace engineer? Inventing a field and snatching up the low hanging fruits doesn't mean somebody would be able to consistently create leading edge output. Most of the advances in deep learning are due to hardware scaling, and the success of a few very specific architectures. Yes credit's due where credit's due, but academia name recognition is very much winner take all. For all the criticism Schumidhuber has received, he has a point. The authors of Attention is all you need, the transformers paper, yolo, have nowhere close to the name recognition of the Turing award trio despite generating similar if not more value through their ideas.
[ Parent
| on HN ]
Re:Geoffrey Hinton leaves Google and can now speak freely ab...
by cma on Monday May 01, 2023 @10:37AM
(#35773364)
> The authors of Attention is all you need, the transformers paper, y
Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before:
Re:Geoffrey Hinton leaves Google and can now speak freely ab...
by mardifoufs on Monday May 01, 2023 @10:40AM
(#35773405)
Schmidhuber claims to have invented a lot of things. It's almost a running gag at this point.
Re:Geoffrey Hinton leaves Google and can now speak freely ab...
by uoaei on Monday May 01, 2023 @11:26AM
(#35774140)
And yet somehow his claims always bear some truth. I understand the comments about boys crying wolf, but it's hard to ignore the facts on the ground.
Re:Geoffrey Hinton leaves Google and can now speak freely ab...
by caycep on Monday May 01, 2023 @02:16PM
(#35776879)
not having a PHD in ML, it's hard for me to evaluate his claims, but how valid are all the obscure papers that he brings up? Did someone actually invent backprop in 1930 in some random corner of the former Soviet Union? Or is it a case of "true but misses the point"?
Re:Geoffrey Hinton leaves Google and can now speak freely ab...
by uoaei on Monday May 01, 2023 @02:47PM
(#35777289)
Often it is indeed the latter, although it is interesting that sometimes despite that it gets at the core of our contemporary understanding of the concepts in question.
Re:Geoffrey Hinton leaves Google and can now speak freely ab...
by telotortium on Monday May 01, 2023 @02:44PM
(#35777263)
"Formal equivalence" means very little for engineering, to be frank - the implementation is the important thing. If I wanted to be snarky, I'd say that neural networks are "formally equivalent" to Fourier analysis, which is 200 years old. I see that the paper proposes an implementation of linearized attention as well, which many others have done, but none of which seem to have caught on (although FlashAttention at least makes attention O(n) in memory, if not computation).
Re:Geoffrey Hinton leaves Google and can now speak freely ab...
by innagadadavida on Monday May 01, 2023 @11:40AM
(#35774388)
There are multiple dimensions here - fame and fortune at the very least and whether it is localized or global in scope.
It is still winner takes all, but if you look at the overall landscape, there are plenty of opportunities where you can have an outsized impact - you can have localized fame and fortune (anyone with AI expertise under their belt have no problems with fortune!)