News for hackers. Stuff that matters.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by jimfleming on Wednesday March 27, 2019 @01:07PM (#19503063)
Highway connections are actually used heavily in NLP in combination with LSTMs.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by slashcom on Wednesday March 27, 2019 @06:04PM (#19506608)
And replaced by residual connections in transformers, which are absolutely dominating LSTMs now.
Re:Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing ... by stochastic_monk on Wednesday March 27, 2019 @07:48PM (#19507516)
Transformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.