News for hackers. Stuff that matters.

A comment on RWKV: Reinventing RNNs for the Transformer Era. Parent comment

Re:RWKV: Reinventing RNNs for the Transformer Era by xiphias2 on Tuesday May 23, 2023 @05:11AM (#36041768)
I believe the naming is perfect: the word attention clearly maches the intention of the model structure. It’s still clearly missing more efficiency, but we had to have ChatGPT / ChatGPT4 working and used to make the further research directions clear (increasing context length and decreasing hallucinations).
[ Parent | on HN ]
Re:RWKV: Reinventing RNNs for the Transformer Era by pico_creator on Tuesday May 23, 2023 @05:20AM (#36041835)
Haha, yea - naming things is hard

Every-time someone comes up and say "this is not attention, because it does X and not Y"

My response is, ok, what would you call it then? Because no one (including me) seem to to able to find a better term, that fits its use case.

[ Parent | on HN ]