Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ...
by orbifold on Wednesday March 27, 2019 @10:14AM
(#19500946)
Ironically Schmidhuber's lab also came up with a more general version of bypassing http://arxiv.org/abs/1505.00387 to train deep networks (with similar performance), of which the popular ResNet is a special case.
[ Parent
| on HN ]
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ...
by tanilama on Wednesday March 27, 2019 @11:37AM
(#19501957)
But highway network doesn't work as well as ResNet, and ResNet is simpler.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ...
by jimfleming on Wednesday March 27, 2019 @01:07PM
(#19503063)
Highway connections are actually used heavily in NLP in combination with LSTMs.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ...
by slashcom on Wednesday March 27, 2019 @06:04PM
(#19506608)
And replaced by residual connections in transformers, which are absolutely dominating LSTMs now.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ...
by stochastic_monk on Wednesday March 27, 2019 @07:48PM
(#19507516)
Transformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ...
by deepnotderp on Wednesday March 27, 2019 @01:08PM
(#19503075)
Highway networks with scalar gating beats ResNet by a small margin.