But the real hero here is not the LLM, but the training set. It took ages to collect all the knowledge, ideas and methods we put in books. It cost a lot of human effort to provide the data. Without the data we would have nothing. Without GPT we could use RWKV, Mamba, S4, etc and still get similar results. It's the data not the model.
> the intuition that language is central to thought predates LLMs
Language carries AI and humans. The same distribution of language can be the software running in our brains and in LLMs. I think humans act like conditional language models with multi modality and actions. We use language to plan and solve our problem, work together and learn (a lot) from others.
Language itself is an evolutionary system and a self replicator. Its speed is much faster than biology. We've been on the language exponential for millennia, but just now hit the critical mass for LLMs to be possible.
It's not so important that GPT-4 is a 2T weights model, what matters is that it was trained on 13T tokens of human experience and it now "writes like a human". Does that mean humans also learn the same skills GPT-4 has learned from its training set mostly by language as well?
And in the case of windmills the hero is the wind. But the mill is still a fantastic achievement.
qed
Yes, that's their whole thing: moving when the air blows.
Humans can just sit around aimlessly toying with stuff, reading books, etc. They’ll figure out some of these patterns on their own. Whereas, we have to give these things a ton of highly-curated, pre-processed data made by human minds of all kinds. Then, it’s usually 800GB-4TB for the good ones. They’re appear to be not in our league yet as learning machines.
We’ll be able to assess it better as multimodal models come online. We can train them like infants, then children, on books, random observations through cameras, TV, people reading to them, supervised feedback… all the stuff we do with humans. Then see if and how they match up in performance.
You mean to imply that birds have learned from jet fighter designs?
I fail to understand the point you're making.
Besides, with human beings, we have a host of philosophical problems that undermine the neuroscientific presumption that a mechanistic and closed view of the brain can account for mental activity entirely, like the problem of intentionality.
All arguments about AGI aside, a machine was built that writes like a human. Its design is very un-biological in places, so it's tempting to dismiss it. Why not see how deep the rabbit hole goes?
Like all conjectures it just might surprise us. For one, the intuition that language is central to thought predates LLMs, but it's certainly consistent with it.