The people calling him annoying are generally hangers-on trying to display that they're part of the group - people who took a class once 20 years ago and think they need now to come explain.
He's a lovely man, and he's 100% correct to be angry that people are getting rich and famous while not citing him. It's an ethical disaster.
I really wish HN people wouldn't pile onto him this way.
.
> The field is full of huge egos and overly simplified narratives from a singular point of view.
Stop it. Primacy in publishing is a core topic in science. If these people were at universities, and not Google or Facebook, they'd have been fired for what they're doing.
Schmidthuber doesn't actually have a huge ego. He's just saying "guys, stop re-publishing my papers and claiming you came up with it."
You should really be deeply disgusted in Hinton right now, not explaining Schmidthuber's ego. Hinton is being a plagiarist *and he knows it*.
This isn't an accident and it isn't the first time.
Hinton is committing fraud, according to academic doctrine. This is an automatic termination at most universities on Earth, even through tenure.
It's time to stop making excuses for these rich men cherry picking other peoples' work for their and their employers' gain.
Schmidthuber is, if anything, being remarkably calm and polite, for having been robbed blind by megacorporations of credit and wealth for decades.
What are some of the papers he has plagiarized?
> 9. LBH claim ReLUs enabled deep learning to outperform previous methods for object recognition, referring to their GPU-based ImageNet 2012 winner called AlexNet,[GPUCNN4] without mentioning that our earlier groundbreaking deep GPU-based DanNet[GPUCNN1-3,5-8][DAN] did not need ReLUs at all to win 4 earlier object recognition competitions and to achieve superhuman results already in 2011[GPUCNN1-8][R5-6] (see Sec. XIV).
If we click and look into the details, Alexnet won imagenet - a general purpose image recognition dataset. Whereas Dannet worked on specific domains- Chinese handwriting recognition, mitosis etc. So Dannet is not comparable in impact to Alexnet at all. ReLUs are in all complex DNNs now - wouldn't have happened if ReLUs are redundant as implied by Schmidhuber.
https://people.idsia.ch/~juergen/computer-vision-contests-wo...
> 8. LBH devote an extra section to rectified linear units (ReLUs), citing papers of the 2000s by Hinton and his former students, without citing Fukushima who introduced ReLUs in 1969[RELU1-2] (see Sec. XIV).
This is only one of many concrete examples given.
DanNet obviously worked on all kinds of image data, otherwise it would not have won all those competitions before the similar AlexNet. However, the CNN pioneer was Fukushima who introduced CNNs and ReLUs.
You are arguing 100% in bad faith. I specifically cited #9 because it is absurdly unreasonable to counter the entire gish galloping. Your counter argument is to cite #8
You are well aware that not citing an earlier paper with different implementation and results is not plagiarism. There is absolutely no evidence of plagiarism anywhere.
If you drop the word "plagiarism" and replace it with "priority in invention" the allegations still don't stick, as I explained for #8.
It is one thing to say that Schmidhuber did not get due credit, but quite another to call Hinton a plagiarist.
Following up on your logic is absurd, because I can conveniently state that back prop is just the chain rule in differentiation by Newton and everyone else has plagiarized from him. And ReLU was plagiarized by Fukushima from neuroscience researchers.
DNNs are an empirical engineering technique. Priority in proposing a technique is not remarkable. Most techniques like ReLU and back prop are straightforward to develop and understand. What matters is the absolute performance gain over SOTA techniques.
I cannot claim priority over Fazlur Rahman if I had the idea to build skyscrapers using a tubular design. This is not theoretical physics. Building a sky scraper is an engineering problem. You have to actually build the sky scraper to claim victory.
ReLU is an absurdly simple gate. The question revolved around its effectiveness, which was proven by Hintons lab.
The key result is the outstanding performance on imagenet. If Schmidhuber was the actual pioneer, why wasn't he able produce the same results before Hinton?
NN were known to work for hand writing recognition since the 90s (including papers by Hinton). Dannet being able to do it for Chinese characters in 2010s is unremarkable.
When Fukushima published ReLUs in 1969 and CNNs in 1979, there were neither decent computers nor competitions. No excuse for not citing him.
> ReLU is an absurdly simple gate. The question revolved around its effectiveness, which was proven by Hintons lab.
Many good things are simple. They should have cited the creator, no matter how much they profited later from faster computers or novel datasets or the like.
> The key result is the outstanding performance on imagenet. If Schmidhuber was the actual pioneer, why wasn't he able produce the same results before Hinton?
Did his team ever participate in imagenet? Apparently not. He writes about DanNet: "For a while, it enjoyed a monopoly. From 2011 to 2012 it won every contest it entered, winning four of them in a row (15 May 2011, 6 Aug 2011, 1 Mar 2012, 10 Sep 2012)"
> NN were known to work for hand writing recognition since the 90s (including papers by Hinton). Dannet being able to do it for Chinese characters in 2010s is unremarkable.
The remarkable thing is that "DanNet was the first pure deep CNN to win computer vision contests." Before DanNet, other methods won the competitions. DanNet changed that.
However, the CNN pioneer was Fukushima who introduced the CNN architecture and ReLUs. Hinton did not cite him.
> You are well aware that not citing an earlier paper with different implementation and results is not plagiarism. There is absolutely no evidence of plagiarism anywhere.
So what exactly constitutes plagiarism? It's not about good or bad faith, it's about checking who did it first. If you are using building blocks from previous papers, you must cite them. Schmidhuber cites the difference between unintentional [PLAG1] and intentional plagiarism [FAKE2]:
[PLAG1] Oxford's guidance to types of plagiarism (2021). Quote: "Plagiarism may be intentional or reckless, or unintentional."
[FAKE2] L. Stenflo. Intelligent plagiarists are the most dangerous. Nature, vol. 427, p. 777 (Feb 2004). Quote: "What is worse, in my opinion, ..., are cases where scientists rewrite previous findings in different words, purposely hiding the sources of their ideas, and then during subsequent years forcefully claim that they have discovered new phenomena."
More quotes: "If one "re-invents" something that was already known, and only becomes aware of it later, one must at least clarify it later, and correctly give credit in follow-up papers and presentations." ... "And the authors did not cite the prior art - not even in later surveys."
This is crucial. Even later they did not cite the original sources.
> Following up on your logic is absurd, because I can conveniently state that back prop is just the chain rule in differentiation by Newton and everyone else has plagiarized from him.
The paper apparently both anticipated and corrected your claim (it wasn't Newton): "Some claim that "backpropagation is just the chain rule of Leibniz (1676) & L'Hopital (1696)." No, it is the efficient way of applying the chain rule to big networks with differentiable nodes (there are also many inefficient ways of doing this). It was not published until 1970.[BP1]"
[BP1] S. Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, 1970. See chapters 6-7 and FORTRAN code on pages 58-60. PDF. See also BIT 16, 146-160, 1976. Link. The first publication on "modern" backpropagation, also known as the reverse mode of automatic differentiation.
> And ReLU was plagiarized by Fukushima from neuroscience researchers.
Really? Can you prove this? Do you have a reference?
LeCun would probably also be upset if he read about his work in the newspaper, but mis-attributed to someone else.
Also, millions in funding money depend on stuff like this. If LeCun can pretend to be the inventor, he gets the grant money. If Schmidhuber gets the citation he seeks, he can use that to apply for new funding. So it's not just prestige, these citation issues have a large monetary impact, too.
Schmidhuber points out everything I said above. JEPA is old idea, etc. He also points out that the paper is "tabloid" article. Schmidhuber does not deserve being quoted for general ideas because he wrote paper of few of them.
The field is full of huge egos and overly simplified narratives from a singular point of view.
For example, it's true that Hinton was a torchbearer of neural networks for several decades and was off mainstream. That gives him well earned legendary status. But it's also true that other methods worked better until early or mid 2000s and had more rigorous theory behind it (theory behind older methods is still better, nn is still mostly experimenting and wandering). Hinton also had position and funding to do pioneering research and collect students all over the world. I remember reading his product of experts paper as a student in 2002 when I worked with boosting and bagging, so clearly he was not completely out of sight.