Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper

Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by dpcx on Wednesday May 17, 2023 @04:42PM (#35981017)

I appreciate the explanation, but I don't know what junior-dev would understand most of this. I may be just a web developer, but I couldn't understand most of this. I'd still have to read for 30m to grok it all.

[ Parent | on HN ]

Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by throwawaymaths on Wednesday May 17, 2023 @04:51PM (#35981133)

Yeah sorry, it still requires math and probably some exposure to ML basics.

[ Parent | on HN ]

Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by shadowgovt on Wednesday May 17, 2023 @05:04PM (#35981261)

I think one hole in the description for simplicity is that "differentiable" it's not an adjective that applies to hash tables.

Differentiable relative to what? What is (x) in the d(hashtable)/d(x) equation?

[ Parent | on HN ]

Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by lhnz on Wednesday May 17, 2023 @05:33PM (#35981528)

I think it applies because lookups can be done by multiplying one-hot vectors (or masks) with matrices; this is roughly analogous to what happens when we multiply Q with K^T in a self-attention head.

Read this: https://e2eml.school/transformers.html#table_lookup

And then read: https://e2eml.school/transformers.html#attention

Also, multiplication is differentiable as you can calculate a partial derivative of this operation, and the ability to do that is what allows gradient descent optimization via the chain rule.

[ Parent | on HN ]

Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by throwawaymaths on Wednesday May 17, 2023 @05:42PM (#35981635)

That's exactly the point, though! It's surprising. A hashtable is a map from keys to values. Making it differentiable means that a small change in the key also makes a small change in the value!

[ Parent | on HN ]

Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by moultano on Wednesday May 17, 2023 @09:41PM (#35983315)

differentiable relative to the model parameters. The attention mechanism does store(key(x), value(x)), followed by lookup(query(y)), where key(), value(), lookup(), query() are all designed to be differentiable.

[ Parent | on HN ]