Re:Ask HN: Can someone ELI5 Transformers and the “Attention ...
by lhnz on Wednesday May 17, 2023 @05:33PM
(#35981528)
I think it applies because lookups can be done by multiplying one-hot vectors (or masks) with matrices; this is roughly analogous to what happens when we multiply Q with K^T in a self-attention head.
Also, multiplication is differentiable as you can calculate a partial derivative of this operation, and the ability to do that is what allows gradient descent optimization via the chain rule.
Read this: https://e2eml.school/transformers.html#table_lookup
And then read: https://e2eml.school/transformers.html#attention
Also, multiplication is differentiable as you can calculate a partial derivative of this operation, and the ability to do that is what allows gradient descent optimization via the chain rule.