News for hackers. Stuff that matters.
Re:RWKV Language Model by yshui on Thursday January 02, 2025 @08:57AM (#42574424)
Any autoregressive model can do what you are describing. transformers are generating one token at a time too, not all at once.
Re:RWKV Language Model by whimsicalism on Thursday January 02, 2025 @10:10AM (#42575065)
yes but transformers are much slower than state space models
Re:RWKV Language Model by intalentive on Thursday January 02, 2025 @02:59PM (#42578156)
True but memory requirements grow with sequence length. For recurrent models the memory requirement is constant. This is why I qualified with "low memory".