Re:Transformers Are Graph Neural Networks
by murbard2 on Sunday September 13, 2020 @11:13AM
(#24461354)
> There is absolutely no training signal that tells an RNN to remember something beyond the BPTT horizon. So why would it?
Because they generalize. Char-RNN learn to balance parenthesis separated by a longer distance than the BPTT window because they've learned that counting parenthesis is useful for prediction based on parenthetical statements shorter than the BPTT window.
Because they generalize. Char-RNN learn to balance parenthesis separated by a longer distance than the BPTT window because they've learned that counting parenthesis is useful for prediction based on parenthetical statements shorter than the BPTT window.