Re:RWKV: Reinventing RNNs for the Transformer Era
by matteogrella on Tuesday May 23, 2023 @07:55AM
(#36043026)
As one of the authors, I'd like to clarify: the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined. This parallelization occurs during both the training and inference stages, specifically during the prompt reading process (consider it an "encoding"), right before the generation (or decoding phase).
[ Parent
| on HN ]
Re:RWKV: Reinventing RNNs for the Transformer Era
by RC_ITR on Thursday May 25, 2023 @12:48PM
(#36073023)
How can something recurrent be parallelized?
> the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined.
And sure, this is the core concept of self attention, no?