News for hackers. Stuff that matters.

A comment on 3Blue1Brown: Visualizing Attention, a Transformer's Heart. Parent comment

Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by vanjajaja1 on Sunday April 14, 2024 @11:56PM (#40036939)
at that point what it has is not a representation of the input, its a representation of what the next output could be. ie. its a lossy process and you can't extract what came in the past, only the details relevant to next word prediction

(is my understanding)

[ Parent | on HN ]
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by rollinDyno on Monday April 15, 2024 @01:58AM (#40037457)
If the point was the presentation of only the next token, and predicted tokens were a function of only the preceding token, then the vector of the new token wouldn’t have the information to produce new tokens that kept telling the novel.
[ Parent | on HN ]