Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based...
by solomatov on Thursday January 19, 2023 @08:21PM
(#34448001)
Naive RNN have vanishing gradient, but LSTMs and GRUs are much better in this respect.
[ Parent
| on HN ]
Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based...
by jszymborski on Friday January 20, 2023 @01:20AM
(#34449798)
While this is true, and was a major advantage of LSTMs/GRUs, they still suffer from vanishing gradients.
w.r.t proteins, our sequences often surpass 1500 amino acids and that is really tough for an LSTM to stably train on.