Revealing example of self-attention, the building block of transformer AI models

Re:Revealing example of self-attention, the building block o... by dwheeler on Saturday April 29, 2023 @08:00PM (#35758489)

For context: The latest large language models (LLM) skip doing some complex things that previous good models did, and focus on attention. See:

"Attention is all you need" https://arxiv.org/abs/1706.03762

[ Parent | on HN ]

Re:Revealing example of self-attention, the building block o... by btown on Saturday April 29, 2023 @08:29PM (#35758634)

This MIT lecture on recurrent neural networks provides a lot of context on why attention was so groundbreaking, and how things were solved before: https://www.youtube.com/watch?v=ySEx_Bqxvvo

And for the Attention Is All You Need paper itself, Andrej Karpathy's "GPT From Scratch" talk is a really great walkthrough of how to make this work in practice, all condensed into two hours of live coding: https://www.youtube.com/watch?v=kCc8FmEb1nY

[ Parent | on HN ]

Re:Revealing example of self-attention, the building block o... by hn_throwaway_99 on Saturday April 29, 2023 @09:20PM (#35758873)

That whole MIT course, Intro to Deep Learning, is great and a really nice introduction to the field.

[ Parent | on HN ]