News for hackers. Stuff that matters.

A comment on RWKV: Reinventing RNNs for the Transformer Era. Parent comment

Re:RWKV: Reinventing RNNs for the Transformer Era by pico_creator on Monday May 22, 2023 @11:39PM (#36039520)
One of the biggest issue with incredibly large context sizes is the lack of training / dataset meant specifically to use such large context sizes.

And this apply to all models. So when a specific model now does badly even at 32k or 50k it’s hard to say if it’s an architecture design issue, or a dataset issue

[ Parent | on HN ]
Re:RWKV: Reinventing RNNs for the Transformer Era by ofermend on Tuesday May 23, 2023 @09:39AM (#36044103)
Yes; and the amount of training examples that are really long (relative to the other training examples) becomes small really fast, so it's also a problem of the long-tail in a sense.
[ Parent | on HN ]
Re:RWKV: Reinventing RNNs for the Transformer Era by mike_hearn on Tuesday May 23, 2023 @10:49AM (#36045117)
It feels like this is where training on code is going to go from important to critical. Most human texts won't require you to look back 100,000 tokens to understand what it means, but if you dump 1000 source files one after the other than it will.
[ Parent | on HN ]