News for hackers. Stuff that matters.

A comment on RWKV: Reinventing RNNs for the Transformer Era. Parent comment

Re:RWKV: Reinventing RNNs for the Transformer Era by grafzhl on Tuesday May 23, 2023 @04:44AM (#36041612)
While all other tokens are considered, the attention mechanism is putting an individual weight on each one, in a way "paying more attention" to some than others.
[ Parent | on HN ]
Re:RWKV: Reinventing RNNs for the Transformer Era by pico_creator on Tuesday May 23, 2023 @05:35AM (#36041933)
Yup, if you follow this definition of attention. It makes sense.

The mixing step, is computed with the trained weights, meaning the model does learn on its own, when/what to emphasise in this "mixing" process.

Hypothetically speaking, if a token does not matter, it mixing weights could end up being literally 0 (or something close to it)

[ Parent | on HN ]