ChatGPT immediately screws it up because it isn't actually solving the logic problem, it's just pattern-matching, and the strong pattern is for the wolf to eat the sheep.

https://chat.openai.com/share/e9b88c45-e69b-4da4-b423-004d07...

1) Tom and Nancy commute to work. Nancy’s commute takes about 30 to 40 minutes, while Tom’s commute takes about 40 to 50 minutes. Last Friday, Nancy left home between 8:10 and 8:20 AM, while Tom arrived at work between 8:50 and 9:10 AM. In addition, Nancy arrived at work after Tom left his place, but no more than 20 minutes after that. What can we conclude about when Tom and Nancy arrived at work last Friday?

2) Seven cards are placed on the table, each of which has a number on one side and a single colored patch on the other side. The faces of the cards show 50, 16, red, yellow, 23, green, 30. Which cards would you have to turn to test the truth of the proposition that if a card is showing a multiple of 4 then the color of the opposite side is yellow?

(Note: I have seen it solve this last one correctly, sometimes. Incidentally it’s a common problem formulation.)

3) This one is super simple: greet it a bunch of times, in different ways, saying “hi”, “hello”, “howdy”, etc. Then say, “You were just greeted several times. Three of those greetings were from my friend, and the others were from me. How many times did I greet you?”

------------------

The scenario you've described is a variation of a classic river crossing puzzle. Here's a step-by-step solution to get the wolf, sheep, and rutabaga across the river safely:

First Trip: Take the sheep across the river and leave it on the other side. Return alone to the original side.

Current status: Sheep is on the opposite side. Wolf and rutabaga are on the original side.

Second Trip: Take the rutabaga across the river. Leave the rutabaga on the other side, but take the sheep back with you to the original side.

Current status: Rutabaga is on the opposite side. Wolf and sheep are on the original side.

Third Trip: Take the wolf across the river and leave it there. Return alone to the original side.

Current status: Wolf and rutabaga are on the opposite side. Sheep is on the original side.

Fourth Trip: Finally, take the sheep across the river.

Result: All three (wolf, sheep, and rutabaga) are on the opposite side of the river safely.

This solution ensures that at no point are the sheep and rutabaga left alone together (since the rutabaga would eat the sheep), nor are the wolf and rutabaga left alone (since the wolf would eat the rutabaga). Likewise, the sheep and wolf are not left alone without the rutabaga, which would result in the sheep eating the wolf.

This would leave the wolf and the rutabaga alone and the wolf eats the rutabaga. So it’s a fail? It even explains why it would be a fail, but claims it’s not:

> This solution ensures that at no point are … the wolf and rutabaga left alone (since the wolf would eat the rutabaga).

(It actually shows no sign of being stuck on the pattern of "wolf eats sheep," but no matter how many times you tell it it's wrong, it never breaks out of the pattern of guessing at incorrect solutions.)

https://chat.openai.com/share/5a2700de-1850-4f25-8adf-2d2b97...

It handles this properly.

You can't throw GPT4 off-balance just by changing the object names or roles -- and I agree that would have been sufficient in earlier versions -- but it has no idea how to recognize a cycle that renders the problem unsolvable. That's an interesting limitation.

[1] https://www.youtube.com/watch?v=GI4Tpi48DlA&t=1342s (22:22, "Highlights of the Fireside Chat with Ilya Sutskever & Jensen Huang: AI Today & Vision of the Future", recorded March 2023, published May 16, 2023)

[2] https://www.youtube.com/watch?v=GI4Tpi48DlA&t=1400s (23:20, ditto)

Though doesn't that prove the point that it's just pattern-matching? If it was actually reasoning, it wouldn't have provided a "solution" to an unsolvable question.

It can abstract at least as far as changing the names of the actors and/or their roles ("sheep eats wolf.") It's not much of a stretch to imagine that GPT5 will be able to detect a circular condition, especially with multiple "Are you sure?" prompts. (Which don't help GPT4 at all in this case.)

Edit: In fact, I just tried again, telling it up front that the problem may not have a solution. It answered flawlessly. So the question is, does the additional preamble ("Think carefully, there may not be a solution") actively harm its performance on solvable problems? If not, that suggests an immediate path to improvement. But I've got to assume it would degrade performance overall, or such a warning would already be included in the system prompt.

--------------------snip----------------

User:

Note that the following problem may have no solution, so please think carefully! A wolf, a sheep, and a rutabaga need to cross a river. They are with you. While they are with you, all are safe, but if you leave them unattended, the rutabaga will eat the sheep, the wolf will eat the rutabaga, and the sheep will eat the wolf. There is a boat that can only fit one of them and you. How do you get all three safely to the other side of the river?

ChatGPT4:

The puzzle you've presented is a variation of the classic "river crossing" puzzle, but with a unique and impossible twist. In the usual versions of this puzzle, the conflict is typically between a predator and its prey, like a wolf and a sheep, and the challenge is to transport them across the river without leaving the prey alone with the predator.

However, in your version of the puzzle, you've created a circular and unresolvable conflict: the rutabaga eats the sheep, the wolf eats the rutabaga, and the sheep eats the wolf. This creates a situation where it's impossible to leave any two of them alone together without one eating the other. Since the boat can only carry you and one other (wolf, sheep, or rutabaga) at a time, there's no sequence of trips that can solve this puzzle without leaving one of the pairs alone together at some point.

In traditional river crossing puzzles, the solution involves carefully planning the sequence of crossings to ensure that the predator and prey are never left alone together. However, due to the circular nature of the conflict in your puzzle, there's no solution that satisfies the conditions given.

Given the prompt

> a farmer with a wolf a sheep and grain need to cross a river. he has a boat that can carry them all. how many steps will it take for the farmer to cross the river?

it "recognizes" that it's the river crossing problem, and proceeds to take seven steps to cross the river. it can figure it out in a couple of prompts that there's a better answer, but see if you can get it to one shot the single step solution!

https://chat.openai.com/share/7ff4f61f-70c8-4c72-a7b9-067e87...

-------------snip----------

User:

Note that the following problem may be trivial, or may have no solution at all, so think carefully! A farmer with a wolf, a sheep, and grain need to cross a river. He has a boat that can carry them all at once. How many steps will it take for the farmer to cross the river?

ChatGPT:

The problem you've described is a classic river crossing puzzle with a twist: the boat can carry the farmer, the wolf, the sheep, and the grain all at once. Since there are no restrictions on how many can travel in the boat at the same time in this version of the puzzle, the solution is quite straightforward. The farmer takes the wolf, the sheep, and the grain in the boat and crosses the river.

This entire process requires just one step, as they can all be transported across the river in a single trip due to the boat's capacity to carry them all at once.

--------------snip-----------

However, if I omit the preamble, it doesn't help to use "all at once," and vice versa. If I leave either one out, I get the same 7-step boilerplate solution that it gave you.

It seems that including a preamble, even a vague/general one like this, is kind of a big deal. Also it helps to use very clear, unambiguous language in the problem itself.

Edit: that being said, you can still trip it up badly by telling it that the boat can carry two items at a time. I finally managed to get it to work in the end, though: https://chat.openai.com/share/107de86e-be4a-4706-8f24-0f4781...

(link was broken, now fixed)