Understanding and Coding the Self-Attention Mechanism of Large Language Models

Re:Understanding and Coding the Self-Attention Mechanism of ... by qigtyofhp on Friday February 10, 2023 @10:17PM (#34749235)

The attention mechanism started as a simple trick to not use recurrent neural networks.

Read the intro in the original paper "Attention is all you need paper" (https://arxiv.org/abs/1706.03762)

This video explains the drawbacks to RNNs and how transformers solve that: https://youtu.be/S27pHKBEp30?t=394

Andrej Karpathy explains attention here: https://youtu.be/kCc8FmEb1nY?t=3719

He explains how attention is seen as a communication network: https://youtu.be/kCc8FmEb1nY?t=4298

[ Parent | on HN ]

Re:Understanding and Coding the Self-Attention Mechanism of ... by godelski on Saturday February 11, 2023 @01:24AM (#34750080)

> Read the intro in the original paper "Attention is all you need"

I wouldn't call this the original "attention" paper. Definitely not the first paper to use the phrase. If you want clear proof of this, let's read the paper

> Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences.

I do think a lot of people's lack of understanding of attention is because they are so focused on DP(S)A that they miss a lot of the broader picture. And math. Not enough people dig into the math.

[ Parent | on HN ]