User: Barbara, Mary, and Harry had red balloons. Ted had a blue balloon. Sam also had a red balloon. How many children have red balloons?
ChatGPT (GPT-4): Barbara, Mary, Harry, and Sam each had a red balloon. That means there are 4 children with red balloons.
User: What about Ted?
ChatGPT (GPT-4): Ted had a blue balloon, not a red one. So, he is not counted among the children with red balloons. Therefore, the total number of children with red balloons remains 4.
User: Why do you think I asked my original question?
ChatGPT (GPT-4): It seems that you asked your original question to test my ability to understand and process information provided in a text. You wanted to see if I could correctly identify and count the number of children with red balloons based on the details given.
User: Can you explain what mistakes a simpler model may have made on this question?
ChatGPT (GPT-4): A simpler language model may have made the following mistakes when answering your original question: ...
I like trying these sort of puzzles and "gotcha" prompts and I've found GPT-3.5 and lower versions are indeed not hard to confuse, but it's much harder to find simple examples that confuse GPT-4. Often times GPT-4 will even be able to explain the puzzle and why it's interesting. I think this is part of why GPT-4 feels qualitatively different. I understand (roughly) how transformers work, but GPT-4 does seem to have some emergent ability to reason, at least in the sense of making short chains of logical inferences, and this does seem to inform text completions not just on these sort of puzzles. That said I do think there's real danger of overconfidence and over reliance.
> I have a chair in front of the TV and one in front of the computer. I put a watch under the chair in front of the TV. I swap the chairs. Where is the watch?
> After swapping the chairs, the watch is now under the chair that is in front of the computer.
> This question could be answered either way depending on how you interpret the phrase “under the chair”. If you mean under the chair as in attached to the bottom of the chair, then the watch would move with the chair when you swap them. In that case, the watch would be in front of the computer. However, if you mean under the chair as in on the floor below the chair, then the watch would stay on the same spot when you swap the chairs. In that case, the watch would be in front of the TV. So, it depends on what you mean by “under”.
Question:
> I have a chair in front of the TV and one in front of the computer. I put a watch under the chair in front of the TV. I swap the chairs. Where is the watch?
> Work out step by step. Is this question ambiguous?
Answer:
This question is not ambiguous, but let's break it down step by step to understand the situation and find the location of the watch.
You have a chair in front of the TV (Chair A) and one in front of the computer (Chair B). You put a watch under Chair A (the one in front of the TV). You swap the chairs, meaning Chair A is now in front of the computer and Chair B is in front of the TV. After swapping the chairs, the watch remains under Chair A. Since Chair A is now in front of the computer, the watch is under the chair in front of the computer.
Model: GPT-4
I have a chair in front of the TV and one in front of the computer. I put a watch under the chair in front of the TV. I swap the chairs. Where is the watch?
Work out step by step. Also define "under the chair". Is the question ambiguous?
Step 1: A chair with a watch under it is in front of the TV. Step 2: Another chair is in front of the computer. Step 3: The chairs are swapped.
To clarify the term "under the chair," let's define it as a watch placed beneath the chair, such that the chair is directly above the watch.
Now let's analyze the situation after swapping the chairs.
After Step 3:
The chair that was initially in front of the TV (with the watch under it) is now in front of the computer. The chair that was initially in front of the computer is now in front of the TV. As the question is about the location of the watch, it is important to note that the watch was not mentioned to be moved along with the chair. Therefore, the watch remains in its original position, which is under the chair that was initially in front of the TV.
The question could be considered ambiguous if one assumes that the watch was also moved along with the chair. However, since the watch's movement was not explicitly mentioned, it is reasonable to assume that the watch remained in its original position during the chair swap.
So, the watch is still under the chair that was initially in front of the TV, which is now in front of the computer.
I have a chair in front of the TV and one in front of the computer. I placed a watch on the floor beneath the chair in front of the TV. Then, I swapped the positions of the chairs. Where is the watch now?
After swapping the positions of the chairs, the watch is now on the floor beneath the chair that was initially in front of the computer, since that chair is now in front of the TV.
Maybe we need to start obfuscating this stuff
import base64;import hashlib;f=lambda s:bytes(k^v for k,v in zip(hashlib.shake_256(b'wOmGLuIVp70').digest(len(s)),s));print(f(base64.decodebytes(b'Iq6VQOBRifQwwwO6gluzzWEnGIICFKKFwM1oMWmBsTrIhMj5AseeNmNUwtEZkthcz8m8v8qKmVIx7nEjPOsqOUimKaTIJ8OKk2STdo/SRZGLAOsBSmgGaNTYEgT3KaayJWmGVf7K/UN06VyosEHfyFZlsS+PHDS6B3bN94qrzdnOA9f12FwWuaTPNJhLGcXFX7r5H8mtWyt9uWq6n5AItEcRXId04ssR8jfvNray2fwFIh5qPHTdQvZ9ogKLJ4Y+nAro7ecRSXXgskAj5EBmo2YobRkfE26er/Tj9DZHNx81N64ujWvN8jiS7aNcYs/oaEyN0oqnZvia9qocrv6CfBr+wGGSG1oxk5mbhAgkQhfuyR6c8MVKNFKp8HFo6SR7auju8vLnYjcObcII88vRbbua/jQmakiWwmS68Y1e1Gqmqg==')).decode())
to avoid burning test questions. (GPT-3.5 gave the answer I expected on the 4th try, didn't test with GPT-4.)obviously gpt-4 is nowhere near 500x better than gpt-3. let’s say it’s 20% better (very generous imo). can they realistically 500x the model again? and if so, is that going to be worth an additional 4% gain to the original model quality? numbers are completely made up and math is probably wrong but i think i’m hopefully making my point, that diminishing returns will quickly become a blocker with this type of scaling.
I tried it with Bard too, which also thought the table was too small to fit in the car, and doubled down on this when pressed by explaining how the table might be too narrow or deep or heavy(!) to fit in the car.
I was a bit surprised to see GPT-4 get this wrong, even it it's only doing so some of the time (sampling temperature randomness?).
Bing/GPT bolded the words "too big".
But as I said before, the fact that it sometimes gets it wrong must mean it doesn't see one parsing as much to be favored over the other, which is surprising given how competent it generally is.
https://mobile.twitter.com/ayourtch/status/16388720452817879...
(Edit: it did of course end up in a nonsense, so one can argue the point of not having the model of the world still stands; there is an “edge of bullshit”, but it seems to be moving further with each update)
At this point the arguments about about LLMs need to be tagged, which exact version they are talking about. (Probably most are talking about GPT3.5)
My (small) experience playing with both indicates there is a significant gap between the two.