Re:Theory of Mind May Have Spontaneously Emerged in Large La...
by simonh on Thursday February 09, 2023 @07:36PM
(#34733942)
We can probe issues like what do language models know and what they understand in several ways. One is through an understanding of the process it’s following. Another is through seeing how that leads to its responses. Then thirdly by looking at the kinds of errors it makes. Using multiple axes of approach like this we can triangulate in on what it’s doing and what it understands.
In terms of how it works, that’s well known and hardly worth repeating in depth, but to summarise it calculates a probability for the next word in a sequence based on a massive training set of human language word sequences.
So what kind of output do they produce? If you ask what it likes to do on the weekend, GPT3 will say generally something about how it likes to spend time with family and friends, because that’s what it has in it’s training set. GPT3 doesn’t have a family, or friends, it doesn’t hang out. It talks about itself because its training set includes people talking about themselves, but it has no concept of self or what it is. It’s a text generator function. It can write a poem about the warm sun on its face, but it doesn't have a face or feel the sun. It’s just regurgitating stuff people wrote about that.
Newer systems like ChatGPT have guard rail functions that catch things like this and say it’s a language model, but the guard rails don’t change the nature of what it is, they’re just overrides little smarter than if, then, else.
So what kind of errors do they make? They can be trivially tricked into talking utter nonsense, or say sensible things in absurd contexts. Here’s an example where someone asked ChatGPT if it spoke Danish, and it replied that no it can’t speak Danish, it’s an English language model , etc. except here’s the kicker, it gave the reply in perfect Danish.
Again they’ve now added guard rails for this failure mode as well. Nevertheless the basic problem persists in the architecture. It’s doesn’t have a clue what anything means, beyond calculating word probabilities. This means if you know how they work, you can craft text prompts that expose how ludicrously unaware they are. This ability to expose their weaknesses demonstrates that we do genuinely understand how they function and what their limitations are.
So I agree yours is a very reasonable question and it’s not trivial to answer satisfactorily. As the guard rails become more complete the failure modes will get harder up find, but they’re still there in the core implementation, they’re just being papered over. There’s not going to be a simple answer. We need to look deeper at the mechanisms and functions of these things. The same goes for human brains of course, we’re just scratching the surface of those too.
In terms of how it works, that’s well known and hardly worth repeating in depth, but to summarise it calculates a probability for the next word in a sequence based on a massive training set of human language word sequences.
So what kind of output do they produce? If you ask what it likes to do on the weekend, GPT3 will say generally something about how it likes to spend time with family and friends, because that’s what it has in it’s training set. GPT3 doesn’t have a family, or friends, it doesn’t hang out. It talks about itself because its training set includes people talking about themselves, but it has no concept of self or what it is. It’s a text generator function. It can write a poem about the warm sun on its face, but it doesn't have a face or feel the sun. It’s just regurgitating stuff people wrote about that.
Newer systems like ChatGPT have guard rail functions that catch things like this and say it’s a language model, but the guard rails don’t change the nature of what it is, they’re just overrides little smarter than if, then, else.
So what kind of errors do they make? They can be trivially tricked into talking utter nonsense, or say sensible things in absurd contexts. Here’s an example where someone asked ChatGPT if it spoke Danish, and it replied that no it can’t speak Danish, it’s an English language model , etc. except here’s the kicker, it gave the reply in perfect Danish.
https://www.reddit.com/r/GPT3/comments/zb4msc/speaking_to_ch...
Again they’ve now added guard rails for this failure mode as well. Nevertheless the basic problem persists in the architecture. It’s doesn’t have a clue what anything means, beyond calculating word probabilities. This means if you know how they work, you can craft text prompts that expose how ludicrously unaware they are. This ability to expose their weaknesses demonstrates that we do genuinely understand how they function and what their limitations are.
So I agree yours is a very reasonable question and it’s not trivial to answer satisfactorily. As the guard rails become more complete the failure modes will get harder up find, but they’re still there in the core implementation, they’re just being papered over. There’s not going to be a simple answer. We need to look deeper at the mechanisms and functions of these things. The same goes for human brains of course, we’re just scratching the surface of those too.