Chain-of-Thought Can Hurt Performance on Tasks Where Thinking Makes Humans Worse

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by Workaccount2 on Wednesday October 30, 2024 @08:21PM (#42002137)

This is a relatively trivial task for current top models.

More challenging are unconventional story structures, like a mom named Matthew with a son named Mary and a daughter named William, who is Matthew's daughter?

But even these can still be done by the best models. And it is very unlikely there is much if any training data that's like this.

[ Parent | on HN ]

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by alexwebb2 on Wednesday October 30, 2024 @08:38PM (#42002235)

That's a neat example problem, thanks for sharing!

For anyone curious: https://chatgpt.com/share/6722d130-8ce4-800d-bf7e-c1891dfdf7...

> Based on traditional naming conventions, it seems that the names might have been switched in this scenario. However, based purely on your setup:

> Matthew has a daughter named William and a son named Mary.

> So, Matthew's daughter is William.

[ Parent | on HN ]

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by rileymat2 on Wednesday October 30, 2024 @11:15PM (#42003142)

How do people fair on unconventional structures? I am reminded of that old riddle involving a the mother being the doctor after a car crash.

[ Parent | on HN ]

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by adwn on Thursday October 31, 2024 @03:31AM (#42004270)

No idea why you've been downvoted, because that's a relevant and true comment. A more complex example would be the Monty Hall problem [1], for which even some very intelligent people will intuitively give the wrong answer, whereas symbolic reasoning (or Monte Carlo simulations) leads to the right conclusion.

[1] https://en.wikipedia.org/wiki/Monty_Hall_problem

[ Parent | on HN ]