News for hackers. Stuff that matters.
Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based... by solomatov on Thursday January 19, 2023 @08:21PM (#34448001)
Naive RNN have vanishing gradient, but LSTMs and GRUs are much better in this respect.
Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based... by jszymborski on Friday January 20, 2023 @01:20AM (#34449798)
While this is true, and was a major advantage of LSTMs/GRUs, they still suffer from vanishing gradients.

w.r.t proteins, our sequences often surpass 1500 amino acids and that is really tough for an LSTM to stably train on.