News for hackers. Stuff that matters.
Re:Cosma Shalizi on Attention and Transformers by Me1000 on Sunday December 24, 2023 @07:45PM (#38758517)
Not OP and have no insight, but the thing that caused it to click for me was when I heard “this token attends to that token”. Basically, there’s a new value created that represents how much one thing (in an LLM its tokens) cares about another thing.

Saying “attends to” vs “attention” help clarify (for me) the mechanics of what’s going on.

There are no comments on this comment yet.