Re:RWKV Language Model
by yshui on Thursday January 02, 2025 @08:57AM
(#42574424)
Any autoregressive model can do what you are describing. transformers are generating one token at a time too, not all at once.
[ Parent
| on HN ]
Re:RWKV Language Model
by whimsicalism on Thursday January 02, 2025 @10:10AM
(#42575065)
yes but transformers are much slower than state space models
Re:RWKV Language Model
by intalentive on Thursday January 02, 2025 @02:59PM
(#42578156)
True but memory requirements grow with sequence length. For recurrent models the memory requirement is constant. This is why I qualified with "low memory".