Re:Ask HN: Can someone ELI5 Transformers and the “Attention ...
by Sai_ on Wednesday May 17, 2023 @01:21PM
(#35978429)
Enlightening example of having a conversation. Makes thing clearer.
Re:Ask HN: Can someone ELI5 Transformers and the “Attention ...
by tikkun on Wednesday May 17, 2023 @02:18PM
(#35979213)
Did you use GPT to write this? (Not a bad thing! It's a decent answer)
Re:Ask HN: Can someone ELI5 Transformers and the “Attention ...
by jasonjmcghee on Wednesday May 17, 2023 @03:05PM
(#35979754)
Immediately thought this was gpt as well.
Assuming prompt was "Explain Transformers and 'Attention is all you need' in a simple way"
Re:Ask HN: Can someone ELI5 Transformers and the “Attention ...
by mcbuilder on Wednesday May 17, 2023 @05:29PM
(#35981482)
I copy pasted the attention is all you need paper into ChatGPT4 and gave it the prompt "Explain like I'm 5 years old".
The Transformer is a new type of computer program that helps translate languages and understand sentences. It works by paying attention to different parts of a sentence at the same time, instead of looking at one word after another like older programs. This makes it faster and better at understanding complicated sentences. It has been tested on translating English to German and English to French and did a really good job.
(Edit)
Re:Ask HN: Can someone ELI5 Transformers and the “Attention ...
by drexlspivey on Wednesday May 17, 2023 @02:20PM
(#35979244)
AFAIK Transformers and context size are orthogonal concepts. You could have large token contexts before. The transformer directs the “attention” to a specific word/token inside the context.
Think of a conversation you had with a friend. While they were talking, you were probably not just listening to the words they were saying right now, but also remembering what they said a few minutes ago. Your brain was connecting the dots between different parts of the conversation to understand the full meaning. Now, imagine if you could only understand each word in isolation and couldn't remember anything from a few seconds ago. Conversations would be pretty hard to understand, right?
In early NLP models, this was a big problem. They couldn't easily look at the "context" of a conversation or a sentence. They could only look at a few words at a time, so they were a bit like our forgetful person. They were good at understanding the meaning of individual words, but not so good at understanding how those words fit together to create meaning.