News for hackers. Stuff that matters.
Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by lhnz on Wednesday May 17, 2023 @05:33PM (#35981528)
I think it applies because lookups can be done by multiplying one-hot vectors (or masks) with matrices; this is roughly analogous to what happens when we multiply Q with K^T in a self-attention head.

Read this: https://e2eml.school/transformers.html#table_lookup

And then read: https://e2eml.school/transformers.html#attention

Also, multiplication is differentiable as you can calculate a partial derivative of this operation, and the ability to do that is what allows gradient descent optimization via the chain rule.

There are no comments on this comment yet.