And in the case of windmills the hero is the wind. But the mill is still a fantastic achievement.
qed
Yes, that's their whole thing: moving when the air blows.
Humans can just sit around aimlessly toying with stuff, reading books, etc. They’ll figure out some of these patterns on their own. Whereas, we have to give these things a ton of highly-curated, pre-processed data made by human minds of all kinds. Then, it’s usually 800GB-4TB for the good ones. They’re appear to be not in our league yet as learning machines.
We’ll be able to assess it better as multimodal models come online. We can train them like infants, then children, on books, random observations through cameras, TV, people reading to them, supervised feedback… all the stuff we do with humans. Then see if and how they match up in performance.
But the real hero here is not the LLM, but the training set. It took ages to collect all the knowledge, ideas and methods we put in books. It cost a lot of human effort to provide the data. Without the data we would have nothing. Without GPT we could use RWKV, Mamba, S4, etc and still get similar results. It's the data not the model.
> the intuition that language is central to thought predates LLMs
Language carries AI and humans. The same distribution of language can be the software running in our brains and in LLMs. I think humans act like conditional language models with multi modality and actions. We use language to plan and solve our problem, work together and learn (a lot) from others.
Language itself is an evolutionary system and a self replicator. Its speed is much faster than biology. We've been on the language exponential for millennia, but just now hit the critical mass for LLMs to be possible.
It's not so important that GPT-4 is a 2T weights model, what matters is that it was trained on 13T tokens of human experience and it now "writes like a human". Does that mean humans also learn the same skills GPT-4 has learned from its training set mostly by language as well?