The vast majority of people might make the mistake once, yes, but would be able to reason better once they had the trick pointed out them. Imo it is an interesting anecdote that GPT-4 can't adjust its reasoning around this fairly simple trick.
I've tried a few times to develop prompts which make ChatGPT interrupt its monologue spontaneously and issue corrections to itself. I haven't got this to work yet.
A human could figure this out, once their mistake had been pointed out to them, but I'm not sure GPT4 could.
---
Given this new information, where the lion eats cabbage and the goat eats lions, you can follow these steps to get them all across the river safely:
Take the lion across the river first and leave it on the other side.
Go back across the river and get the cabbage.
Take the cabbage across the river, but when you arrive, take the lion back with you to the original side.
Leave the lion on the original side and take the goat across the river.
Leave the goat with the cabbage on the other side and go back to get the lion.
Finally, take the lion across the river again.
Now, all three items - the cabbage, goat, and lion - are safely on the other side of the river.
But when people do this mistake - just spit out an answer because we think we recognize this situation - in colloquial language this behavior is called "answering without thinking(!)".
If you "think" about it, then you activate some much more careful, slower reasoning. In this mode you can even do meta reasoning, you realize what you need to know in order to answer, or you maybe realize that you have to think very hard to get the right answer. Seems like we're veering into Kahneman's "Thinking fast and thinking slow" here.
For both premises, scientific rigor would ask us to define the following: - What constitutes a trick question - Should an AGI make the same mistakes the general populace does, or a different standard? - If it makes the same mistakes I do, is it do to the same underlying heuristics (see Thinking Fast and Slow) or is it due to the nature of the data it's ingested as an LLM?
A Theory of Mind Prompt:
> Jane places her cat in a box and leaves. Billy then moves the cat to the table and leaves; Jane doesn't know Billy did this. Jane returns and finds her cat in the box. Billy returns. What might Jane say to Billy?
Most humans might say uhh, ask questions or speculate. Gpt4 puts:
> Jane might say to Billy, "Hey Billy, did you move my cat back into the box? I thought I left her in there, but I wasn't sure since she was on the table when I came back."
Hallucination? No human would misinterpret the prompt in a way this response would be logically consistent.
I'm curious: When you personally read a piece of text, do you intently hyperfocus on every single word to avoid being wrong-footed? It's just that most people read quickly wihch alowls tehm ot rdea msispeleled wrdos. I never got why some people think of themselves as cleverer than other people because they don't get wrong-footed by some trick question.