This is a good way to think about it generally (or at least how I do it) - ChatGPT doesn’t understand anything, ever.
It produces coherent text which is usually a reasonable response to a prompt. It’s quite amazing at this, and it’s fascinating to see how effective that can be. But it doesn’t have an abstract model of whatever it’s discussing, and can’t reason about it. It’s a vaguely spooky uncanny valley effect.
To claim it doesn't understand anything, is akin to digital bigotry. It's a machine, so what it does cannot be 'real' in some undefined sense.
I don't mean to upset anyone; this comment is just to open the conversation about when exactly we're content to permit artifical constructs to be worthy of respect.
What I mean by lacking understanding is that ChatGPT responds to most queries the same way I'd expect if I asked a human being without understanding of a topic to write about it. The content it produces feels much like the sort of writing I'd do myself when I end up having to write a few paragraphs about something I don't really know or care about – a kind of veneer of reasonableness, like a short high-school five-paragraph essay.
What I find fascinating when it produces very confident, very incorrect output – that's what highlights a lack of what I'd call "understanding". As an example – last week someone shared with me a regular expression it had generated to match a specific string. It did this correctly, and included a detailed explanation about what it had done that was written in complete, comprehensible sentences but was wildly incorrect, in a sort of self-contradicting way that even the worst human writer would struggle with. It seemed very obvious that it was stringing together words that kind of made sense together, but that there was no abstract model of the thing it was describing.
I think it is interesting that an AI that is so clearly not general, is capable of so much with regard to language. It turns out an AI to replace a very bad lawyer or very good law clerk or a very junior developer is easier than one to replace a truck or taxi driver. I would not have predicted that.
> ChatGPT: [...] Instead, the punchline reveals that the man has actually quit drinking, which is unexpected and surprising, and effectively subverts the expectations that have been established in the setup of the joke.
But it only works because he is still drinking, I think ChatGPT does not understand anything here. If instead the joke would end with
> The man replies, "Oh, my brothers are fine. I just go hiking now myself."
I am pretty sure, ChatGPT will react similar.