Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based...
by thegeomaster on Thursday January 19, 2023 @06:59PM
(#34447401)
There's already research that tries to fix this problem with transformers in general, like Transformer-XL [1]. I'm a bit puzzled that I don't see much interest in getting a pre-trained model out that uses this architecture---it seems to give good results.
[1]: https://arxiv.org/abs/1901.02860