Re:Input to GPT Matters a Lot
by tjr on Monday April 03, 2023 @11:28AM
(#35425394)
My current opinion is that it's not so much that GPT is utterly amazing, as it is that language is even more amazing than we previously realized, and GPT happens to have been trained on a huge quantity of language material.
[ Parent
| on HN ]
I was just thinking about this, and the implications it could have on our understanding of the origin and development of human cognitive capabilities, both in the evolutionary and developmental psychology senses.
Language, we are learning, happens to correlate extremely well with the actual structure of the world. In hindsight, it’s not that surprising. The entire purpose of language is to help humans send information about the world to each other!
To predict the next word very accurately for a huge variety of texts, a model must have some amount of real understanding. Sure, it may look different than human understanding.
But there’s no way around this. GPT could not predict so accurately if it didn’t have any understanding the world.
These have historically often been thought of separately: knowledge representation was the data, and language was the interface.
To at least some degree, maybe language is enough for both. Language doesn't just express concepts, but it contains them.