News for hackers. Stuff that matters.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by li4ick on Wednesday March 27, 2019 @08:52AM (#19500315)
I think Schmidhuber should be up there. LSTMs are everywhere, not just NLP. The latest Starcraft 2 bot from Deepmind uses LSTMs extensively in its architecture.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by nabla9 on Wednesday March 27, 2019 @09:20AM (#19500529)
Schmidhuber's significance in the field is not at the same level. He invented one type of neural network, but the "Canadian mafia" launched the whole Deep Learning revolution with multiple breakthroughs and theoretical understanding.

LSTM's are clever and they were bleeding edge up to 2014, but once they were understood better as a bypass mechanism, attention, context vectors and averaging networks and causal convolution are starting to replace them.

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by orbifold on Wednesday March 27, 2019 @10:14AM (#19500946)
Ironically Schmidhuber's lab also came up with a more general version of bypassing http://arxiv.org/abs/1505.00387 to train deep networks (with similar performance), of which the popular ResNet is a special case.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by tanilama on Wednesday March 27, 2019 @11:37AM (#19501957)
But highway network doesn't work as well as ResNet, and ResNet is simpler.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by jimfleming on Wednesday March 27, 2019 @01:07PM (#19503063)
Highway connections are actually used heavily in NLP in combination with LSTMs.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by slashcom on Wednesday March 27, 2019 @06:04PM (#19506608)
And replaced by residual connections in transformers, which are absolutely dominating LSTMs now.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by stochastic_monk on Wednesday March 27, 2019 @07:48PM (#19507516)
Transformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by deepnotderp on Wednesday March 27, 2019 @01:08PM (#19503075)
Highway networks with scalar gating beats ResNet by a small margin.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by phowon on Wednesday March 27, 2019 @11:26AM (#19501817)
Putting aside the whole Schmidhuber debate - where are people getting this idea that causal convolutions are anywhere near the prominence of RNNS/LSTMs?

As far as I'm aware, causal convolutions were used in WaveNet (and subsequent models) and a small number of NLP applications. Meanwhile, LSTM-based models are used in just about every NLP paper, and at least a baseline in the newer ones more dominated by Transformers.

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by moultano on Wednesday March 27, 2019 @02:46PM (#19504486)
I think LSTMs have been mostly abandoned. I don't see much new work using them, vs CNNs or transformers.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by phowon on Wednesday March 27, 2019 @02:53PM (#19504570)
Aren't causal convolutions basically CNNs with masking?
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by tassl on Wednesday March 27, 2019 @02:10PM (#19504032)
"Schmidhuber's significance in the field is not at the same level"

Out of curiosity, at the same level as any of the other three? I could see it from Hinton, maybe LeCunn, but Bengio?

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by srean on Wednesday March 27, 2019 @03:52PM (#19505235)
So ... Schmidhuber's was the seminal work and the later ones were incremental improvements that came from an understanding of Schmidhuber's work ?
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by anyBodyCanDoDL on Wednesday March 27, 2019 @01:52PM (#19503754)
That's very ignorant to say that attention or convolution is anywhere close to the expressivity of RNNs(LSTM). Instead of reading a few cherry-picked results from a model with extremely tuned hyperparameters, pick any random set of tasks and experiment yourself. Based on my experience, LSTM always performs better than all unless you brutely search over hundreds of hyperparameters configs.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by probably_wrong on Wednesday March 27, 2019 @02:26PM (#19504256)
> That's very ignorant to say that attention or convolution is anywhere close to the expressivity of RNNs(LSTM).

But that's exactly the point of the Transformer model, with a paper aptly titled "Attention is all you need" [1]. And the Bert architecture, based in this idea, seems to be doing well. And they claim to be bery flexible, too[2].

Maybe that's what you meant with "unless you brutely search over hundreds of hyperparameters configs", but then again, isn't that what NNs are about anyway?

[1] https://arxiv.org/abs/1706.03762

[2] https://arxiv.org/abs/1810.04805

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by phowon on Wednesday March 27, 2019 @03:16PM (#19504857)
The success of Transformers aside, I'm not sure you should be relying on model titles for anything, lest we forget papers like "One Model To Learn Them All" [1].

[1] https://arxiv.org/abs/1706.05137

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by Nimitz14 on Wednesday March 27, 2019 @09:22AM (#19500550)
Schmidhuber did way more than LSTMs. And they still are SOTA on many tasks. You clearly are not very familiar with the field.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by dang on Wednesday March 27, 2019 @01:04PM (#19503046)
Whoa. Personal attacks will get you banned here. We've warned you about this more than once before. Please review https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here. When you do, don't miss this one, since it should have changed your assumption about the GP:

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by Nimitz14 on Wednesday March 27, 2019 @01:41PM (#19503581)
You're right. Sorry.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by srean on Wednesday March 27, 2019 @03:56PM (#19505291)
Just a note, I did not find it any more rude than other comments that get a free pass. Commenting that more familiarity with the literature is necessary can be a very factual claim.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by dang on Wednesday March 27, 2019 @04:50PM (#19505893)
It was an obvious violation of HN's rules. If you see other comments that violate the rules going unmoderated, the likeliest explanation is that we didn't see them; we don't come close to seeing everything that gets posted. In that case the thing to do is to flag the comment (described at https://news.ycombinator.com/newsfaq.html). In egregious cases, you can email [email protected] as well.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by nabla9 on Wednesday March 27, 2019 @11:59AM (#19502236)
I'm not saying that Schmidhuber is not important pioneer in the field.

When you put researchers or groups into order of importance, the work from this trio comes before others, including Schmidhuber. The Turing Award is given to those at the top and it's not inclusive.

If I could give Turing Award for someone in the field who has not received it yet, I would give it to Vladimir Vapnik.

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by gambler on Wednesday March 27, 2019 @05:19PM (#19506155)
>If I could give Turing Award for someone in the field who has not received it yet, I would give it to Vladimir Vapnik.

So would I, but I don't think that will happen. He's is pretty much an antithesis to everything Deep Learning is about. Theory-first over trial-and-error, math over intuition, small datasets over big data, advances in understanding vs advances in results.

I haven't seen a single discussion of his paper on combining classifiers[1] anywhere on the web.

[1] - http://jmlr.csail.mit.edu/papers/volume17/16-137/16-137.pdf

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by Nimitz14 on Wednesday March 27, 2019 @01:36PM (#19503490)
Sorry.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by pault on Wednesday March 27, 2019 @10:08AM (#19500896)
That third sentence is not necessary.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by tanilama on Wednesday March 27, 2019 @11:55AM (#19502195)
If LSTM is up there, maybe Transformer authors should also be listed somewhere? I mean LSTM is fine and all, IIRC, it only becomes popular after one of Hinton's students put it to work? And that is like in 2013. Similar tales to the CNN as well.

Hinton/Lecun pioneered early BP research, which is significant and fundamental. Bengio has attention mechanism/GAN, and in early days greedy layer wise pretraining for RBM (which isn't something end up today, but pertaining is a thing back in pre 2010 era).

Not saying LSTM isn't significant, but his achievements aren't quite there with those 3, looking closer.

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by h8hawk on Wednesday March 27, 2019 @09:51AM (#19500773)
Schmidhuber didn't invent LSTM Just by himself. Original papaer was published by Sepp Hochreiter and Schmidhuber.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by jmmcd on Wednesday March 27, 2019 @10:23AM (#19501023)
Yes, and the vast, vast majority of important papers by the three winners were co-authored in the same way with supervisors, graduate students or other collaborators.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by moultano on Wednesday March 27, 2019 @02:52PM (#19504554)
I think we need a few more years to see whether LSTMs retain their importance over time. Awarding it now seems premature.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by high_derivative on Wednesday March 27, 2019 @09:05AM (#19500414)
Yes seems political to snub him, but then ACM is an American association.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by dirtybird04 on Wednesday March 27, 2019 @10:34AM (#19501158)
umm two of the award recipients are Canadians
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by high_derivative on Wednesday March 27, 2019 @12:03PM (#19502286)
I encourage downvoters to read the article below for a perspective on how LeCun, Hinton and Bengio may have unfairly excluded a wealth of prior work in their version of deep learning history:

http://people.idsia.ch/~juergen/deep-learning-conspiracy.htm...

Note for people put off by the link: The conspiracy refers to the title of a paper by the Turing award winners, not Schmidhuber's.

Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by theWheez on Wednesday March 27, 2019 @10:11AM (#19500923)
Dr. LeCun is a French native, for what it's worth
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by screye on Wednesday March 27, 2019 @10:16AM (#19500966)
But has spent almost all of his professional career in A&T labs, NYU, FB and Toronto.

That's not true with Schmidhuber.