RWKV: Reinventing RNNs for the Transformer Era

Re:RWKV: Reinventing RNNs for the Transformer Era by jokteur on Tuesday May 23, 2023 @02:30AM (#36040652)

But this is becaude ASIOF was in the training dataset. Chatgpt wouldn't be able to say anything about this book if it wasn't in his dataset, and you wouldn't be able to have enough tokens to present the whole book to chatgpt.

[ Parent | on HN ]

Re:RWKV: Reinventing RNNs for the Transformer Era by nl on Tuesday May 23, 2023 @02:45AM (#36040760)

Thinking of it as "the training dataset" vs "the context window" is the wrong way of looking at it.

There's a bunch of prior art for adaption techniques for getting new data into a trained model (fine tuning, RLHF etc). There's no real reason to think there won't be more techniques that turn what think of now as the context window into something that alters the weights in the model and is serialized back to disk.

[ Parent | on HN ]

Re:RWKV: Reinventing RNNs for the Transformer Era by dcl on Tuesday May 23, 2023 @03:43AM (#36041189)

It's a reasonable way to look at it given that's how pretty much all 'deployed' versions of LLM's work?

[ Parent | on HN ]

Re:RWKV: Reinventing RNNs for the Transformer Era by dcl on Tuesday May 23, 2023 @03:38AM (#36041154)

Exactly.

But also, not just ASIOF is in the training set, but presumably lots of discussion about it and all the interesting events in the book.

[ Parent | on HN ]