Haven't fully watched this but from a brief skimming, here are some differences that the book has:

- it implements a real word-level LLM instead of a character-level LLM

- after pretraining also shows how to load pretrained weights

- instruction-finetune that LLM after pretraining

- code the alignment process for the instruction-finetuned LLM

- also show how to finetune the LLM for classification tasks

- the book it overall has a lots of figures. For Chapter 3, there are 26 figures alone :)

The video looks awesome though. I think it's probably a great complementary resource to get a good solid intro because it's just 2 hours. I think reading the book will probably be more like 10 times that time investment.

Thank you for the answer! What is the knowledge that your book requires? If I have a lot of software dev experience and sorta kinda remember algebra from uni, would it be a good fit?

Good question. I think a Python background is strongly recommended. PyTorch knowledge would be a nice to have (although I've written a comprehensive 40 page intro for the Appendix, which is also already available). From a math perspective, I think it should be gentle. I'm introducing dot products in Chapter 3, but I also explain how you could do the same with for-loops. Same with matrix multiplication. I'm bad at estimating requirements, but I hope this should be sufficient.

You can't understand it unless you already know most of the stuff.

I've watched it many times to understand well most of it.

And obviously you must already know pytorch really well, including the matrix multiplication, backpropagation etc. He speaks very fast too...

Did you really watch all videos in the playlist? I am at video 4 and had no background in PyTorch or numpy.

In my opinion he covers everything needed to understand his lectures. Even broadcasting and multidimensional indexing with numpy.

Also in the first lecture you will implement your own python class for building expressions including backprop with an API modeled after PyTorch.

IMHO it is the second lecture I can recommend without hesitation. The other is Gilbert Strang on linear algebra.

To echo this sentiment, I thought he does a really reasonable job of working up to the topic. Sure, it is fast paced, but it is a video you can rewind, plus play with the notebooks.

There is a lot to learn, but I think he touches on all of the highlights which would give the viewer the tools to have a better understanding if they want to explore the topic in depth.

Plus, I like that the videos are not overly polished. He occasionally makes a minor blunder, which really humanizes the experience.

I was talking about the last video. It's difficult unless you don't know most of the material, or if you havent watched the other videos in the series.

Anyway those videos are quite advanced. Surely not for beginners.

He has like 4 or 5 videos that can be watched before that one where all of that is covered. He goes over stuff like writing back prop from scratch and implementing layers without torch.

I know... That material isn't for beginners.

...but then, what material did you expect as a beginner?

How can Karpathy videos defined for beginners when you have to know: programming, python, pytorch, matrix multiplication, derivatives...