RWKV: Reinventing RNNs for the Transformer Era

Re:RWKV: Reinventing RNNs for the Transformer Era by ftxbro on Monday May 22, 2023 @10:30PM (#36039109)

> "Our experiments reveal that RWKV performs on par with similarly sized Transformers"

[ Parent | on HN ]

Re:RWKV: Reinventing RNNs for the Transformer Era by zh217 on Tuesday May 23, 2023 @05:04AM (#36041732)

This sounds pretty bad, right? Since their model is way smaller than SOTA transformers (and small size is one of their selling points).

[ Parent | on HN ]

Re:RWKV: Reinventing RNNs for the Transformer Era by pico_creator on Tuesday May 23, 2023 @05:49AM (#36042005)

The paper is meant to compare architecture vs architecture with similar model size, and dataset - to inform decisions in future architecture designs

Its main benefits being presented with the above staying closely the same, is that it has significantly lower running and training cost without performance penalty

If you want to compare any <20B model with GPT 3.5 / 4 / 100B models evals, thats another paper altogether

[ Parent | on HN ]