Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart
by rcarmo on Monday April 15, 2024 @04:20AM
(#40038054)
You're effectively steering the predictions based on adjacent vectors (and precursors from the prompt). That mental model works fine.
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart
by eurekin on Monday April 15, 2024 @04:47AM
(#40038174)
> TBF there is no good explanation why it works
I thought the general consesus was: "transformers allow neural networks to have adaptive weights".
As opposed to the previous architectures, were every edge connecting two neurons always has the same weight.
EDIT: a good video, where it's actually explained better: https://youtu.be/OFS90-FX6pg?t=750&si=A_HrX1P3TEfFvLay
My mental justification for attention has always been that the output of the transformer is a sequence of new token vectors such that each individual output token vector incorporates contextual information from the surrounding input token vectors. I know it's incomplete, but it's better than nothing at all.