News for hackers. Stuff that matters.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by inciampati on Sunday June 18, 2023 @01:36AM (#36377608)
Recurrent models like RWKV should theoretically allow for unbounded context size. The problem is training them, which requires looking at a lot of long contexts and which isn't well supported by the RWKV "trains like a transformer, runs like an RNN" model.

There are no comments on this comment yet.