News for hackers. Stuff that matters.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by tanilama on Wednesday March 27, 2019 @11:37AM (#19501957)
But highway network doesn't work as well as ResNet, and ResNet is simpler.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by jimfleming on Wednesday March 27, 2019 @01:07PM (#19503063)
Highway connections are actually used heavily in NLP in combination with LSTMs.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by slashcom on Wednesday March 27, 2019 @06:04PM (#19506608)
And replaced by residual connections in transformers, which are absolutely dominating LSTMs now.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by stochastic_monk on Wednesday March 27, 2019 @07:48PM (#19507516)
Transformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by deepnotderp on Wednesday March 27, 2019 @01:08PM (#19503075)
Highway networks with scalar gating beats ResNet by a small margin.