News for hackers. Stuff that matters.
Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based... by thegeomaster on Thursday January 19, 2023 @06:59PM (#34447401)
There's already research that tries to fix this problem with transformers in general, like Transformer-XL [1]. I'm a bit puzzled that I don't see much interest in getting a pre-trained model out that uses this architecture---it seems to give good results.

[1]: https://arxiv.org/abs/1901.02860

Re:ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based... by gok on Thursday January 19, 2023 @08:02PM (#34447861)
T5 uses relative positional encoding