News for hackers. Stuff that matters.

A comment on Eagle 7B: Soaring past Transformers.

Re:Eagle 7B: Soaring past Transformers by _hl_ on Monday January 29, 2024 @01:37AM (#39173416)
Attention is, after all, not what you need.
[ Parent | on HN ]
Re:Eagle 7B: Soaring past Transformers by jeremycochoy on Tuesday January 30, 2024 @05:31PM (#39196839)
Well, RWKV is using some linear (non-quadratic) form of attention so... strictly speaking... you still need a bit of attention :D
[ Parent | on HN ]