News for hackers. Stuff that matters.

A comment on RWKV: Reinventing RNNs for the Transformer Era. Parent comment

Re:RWKV: Reinventing RNNs for the Transformer Era by PoignardAzur on Tuesday May 23, 2023 @07:42AM (#36042928)
> The chinchilla law states that the amount of data required to train a language model grows exponentially with the model size. This means that it is very expensive to train large language models, even with the latest hardware. The RWKV community is working on developing new methods for training large language models more efficiently. There are a number of datasets available to the RWKV community, including:

What? I though the chinchilla-optimal regime was something like "20 tokens per weight". That's not remotely exponential, even by the word's colloquial use.

[ Parent | on HN ]
Re:RWKV: Reinventing RNNs for the Transformer Era by stellaathena on Tuesday May 23, 2023 @09:21AM (#36043876)
It seems that the user is posting ChatGPT-generated text and not a real summary. It's complete nonsense, with about half of the sentences containing a factual error.
[ Parent | on HN ]