The Transformer Family

Re:The Transformer Family by hn_throwaway_99 on Sunday January 29, 2023 @02:41PM (#34571876)

Somewhat off topic. As someone who did some neural network programming in Matlab a couple decades ago, I always feel a bit dismayed that I'm able to understand so little about modern AI given the explosion in advances in the field starting in about the late 00s or so with things like convolutional neural networks and deep learning, transformers, large language models, etc.

Can anyone recommend some great courses or other online resources for getting up to speed on the state-of-the-art with respect to AI? Not really so much looking for an "ELI5" but more of a "you have a strong programming and very-old-school AI background, here are the steps/processes you need to know to understand modern tools".

[ Parent | on HN ]

Re:The Transformer Family by rjtavares on Sunday January 29, 2023 @02:45PM (#34571919)

The obvious answer is Fast AI's Practical Deep Learning for Coders - https://course.fast.ai/

They'll also be releasing the "From Deep Learning Foundations to Stable Diffusion" course soon, which is basically Part 2 of the course.

[ Parent | on HN ]

Re:The Transformer Family by heyitsguay on Sunday January 29, 2023 @02:55PM (#34572007)

For a while now, an answer I've seen is to start with "Attention Is All You Need", the original Transformers paper. It's still pretty good, but over the past year I've led a few working sessions on grokking transformer computational fundamentals and they've turned up some helpful later additions that simplify and clarify what's going on.

You can quickly get overwhelmed by the million good resources out there so I'll keep it to these three. If you have a strong CS background, they'll take you a long way:

(1) Transformers from Scratch: https://peterbloem.nl/blog/transformers

(2) Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547de...

(3) Formal Algorithms for Transformers: https://arxiv.org/abs/2207.09238

[ Parent | on HN ]

Re:The Transformer Family by bfung on Sunday January 29, 2023 @03:37PM (#34572457)

* * *

[ Parent | on HN ]

Re:The Transformer Family by peterfirefly on Sunday January 29, 2023 @05:00PM (#34573241)

Yannic Kilcher is great but this video worked better for me:

"LSTM is dead. Long live transformers!" (Leo Dirac): https://www.youtube.com/watch?v=S27pHKBEp30

[ Parent | on HN ]

Re:The Transformer Family by nerdponx on Sunday January 29, 2023 @03:35PM (#34572439)

* * *

[ Parent | on HN ]

Re:The Transformer Family by peterfirefly on Sunday January 29, 2023 @05:01PM (#34573249)

That problem doesn't really disappear with teachers and classes ;)

[ Parent | on HN ]

Re:The Transformer Family by d4rkp4ttern on Monday January 30, 2023 @08:04AM (#34579069)

I second the recommendation for Peter Bloem’s tutorial.

I’m also about to read the transformer chapter from this excellent upcoming book by Simon Prince:

udlbook https://udlbook.github.io/udlbook/

[ Parent | on HN ]

Re:The Transformer Family by whoateallthepy on Sunday January 29, 2023 @03:17PM (#34572244)

I put together a repository at the end of last year to walk through a basic use of a single layer Transformer: detect whether "a" and "b" are in a sequence of characters. Everything is reproducible, so hopefully helpful at getting used to some of the tooling too!

https://github.com/rstebbing/workshop/tree/main/experiments/...

[ Parent | on HN ]

Re:The Transformer Family by ad404b8a372f2b9 on Sunday January 29, 2023 @03:30PM (#34572380)

http://cs231n.stanford.edu/ is good for convolutional networks.

[ Parent | on HN ]

Re:The Transformer Family by sorenjan on Sunday January 29, 2023 @04:48PM (#34573116)

A course by Andrej Karpathy on building neural networks, from scratch, in code. We start with the basics of backpropagation and build up to modern deep neural networks, like GPT.

https://karpathy.ai/zero-to-hero.html

[ Parent | on HN ]

Re:The Transformer Family by PartiallyTyped on Sunday January 29, 2023 @06:24PM (#34573887)

For something totally different, try “Geometric Deep Learning”. There is a course by the authors of the book.

Once you understand how everything is operations on graphs, most kernels and functions become very easy to understand.

[ Parent | on HN ]

Re:The Transformer Family by yboris on Sunday January 29, 2023 @07:50PM (#34574503)

I think this book is excellent explanation of some architectures, how they work, and how you can quickly build some with Keras

https://www.manning.com/books/deep-learning-with-python-seco...

Written by François Chollet, the creator of Keras

[ Parent | on HN ]

Re:The Transformer Family by rcme on Sunday January 29, 2023 @09:30PM (#34575186)

I think the concepts are simple. After all, everything is just a mutli-variable derivative. However, I find the choice of notation very confusing. Mostly because it's impossible to remember the shape of everything.

Even if this linked post, they have a "notations" section at the top. Almost immediately, they start using a value k that isn't defined anywhere.

[ Parent | on HN ]

Re:The Transformer Family by 0cf8612b2e1e on Monday January 30, 2023 @12:24AM (#34576100)

Not that I think statistics terminology or notation is worthy of praise (it’s mostly horrible), but it frustrates me to no end how the ML world reappropriated terms seemingly just to be different.

[ Parent | on HN ]