In pop culture in 2010+ sure, but he was essentially parroting Feynman IIRC.
"How to learn AI from first principles?"
Start with https://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_t... and eventually you'll get to AI, exercise left to the reader ;)
> The other big question is why you want to learn it.
Good question. I'm just looking for a wider context to understand contemporary AI. I don't know if this serves any practical purpose but I'm someone who likes to understand the "why" behind everything and starting from "first principles" helps uncover that.
I like learning things starting from small, atomic this, then building up and learning higher layers of abstraction and functionality later. I tend to find hands on totals too "top down" in the sense that they start with all the told in place and then give you a cursory look into what's actually happening.
Personally I feel like most things in the world aren't really that complicated when you understand the building blocks. There are a few core ideas and then a bunch of layers on top to organize and utilize those ideas for different applications. So if I have an interest in something I want to learn from the ground up.
The other big question is why you want to learn it. If you want to learn ML in itself, than anything including the search algorithms (which used to be considered core to ML a long time ago) you mentioned is part of that. But if you want to learn ML to contribute to modern developments like LLMs, then search algorithms are virtually useless. If you aren't going to be engineering any ML or ML products, what you want is to gain some insight into it's future and the business of it. So learning things like transformer architecture is going to be far more unhelpful than say, reading about the economics of compute clusters.
Given the empirical/engineering quality of current ML, I'd say building it from scratch is really good for getting the handful of possible first principles (the fundamental functions involved, data cleaning, training, etc)