Re:Cosma Shalizi on Attention and Transformers
by Me1000 on Sunday December 24, 2023 @07:45PM
(#38758517)
Not OP and have no insight, but the thing that caused it to click for me was when I heard “this token attends to that token”. Basically, there’s a new value created that represents how much one thing (in an LLM its tokens) cares about another thing.
Saying “attends to” vs “attention” help clarify (for me) the mechanics of what’s going on.
Saying “attends to” vs “attention” help clarify (for me) the mechanics of what’s going on.