Re:The Secret Sauce behind 100K context window in LLMs: all ...
by inciampati on Sunday June 18, 2023 @01:36AM
(#36377608)
Recurrent models like RWKV should theoretically allow for unbounded context size. The problem is training them, which requires looking at a lot of long contexts and which isn't well supported by the RWKV "trains like a transformer, runs like an RNN" model.
[ Parent
| on HN ]
There are no comments on this comment yet.