Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on RWKV: Reinventing RNNs for the Transformer Era. Parent comment
The mixing step, is computed with the trained weights, meaning the model does learn on its own, when/what to emphasise in this "mixing" process.
Hypothetically speaking, if a token does not matter, it mixing weights could end up being literally 0 (or something close to it)
There are no comments on this comment yet.
The mixing step, is computed with the trained weights, meaning the model does learn on its own, when/what to emphasise in this "mixing" process.
Hypothetically speaking, if a token does not matter, it mixing weights could end up being literally 0 (or something close to it)