News for hackers. Stuff that matters.

A comment on Chain-of-Thought Can Hurt Performance on Tasks Where Thinking Makes Humans Worse. Parent comment

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by danenania on Thursday October 31, 2024 @01:50PM (#42009409)
> Chain of thought is like trying to improve JPG quality by re-compressing it several times. If it's not there it's not there.

Empirically speaking, I have a set of evals with an objective pass/fail result and a prompt. I'm doing codegen, so I'm using syntax linting, tests passing, etc. to determine success. With chain-of-thought included in the prompting, the evals pass at a significantly higher rate. A lot of research has been done demonstrating the same in various domains.

If chain-of-thought can't improve quality, how do you explain the empirical results which appear to contradict you?

[ Parent | on HN ]
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by mplewis on Thursday October 31, 2024 @03:19PM (#42010453)
The empirical results like OP’s paper, in which chain of thought reduces quality?
[ Parent | on HN ]
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by danenania on Thursday October 31, 2024 @03:50PM (#42010774)
The paper is interesting because CoT has been so widely demonstrated as effective. The point is that it "can" hurt performance on a subset of tasks, not that CoT doesn't work at all.

It's literally in the second line of the abstract: "While CoT has been shown to improve performance across many tasks..."

[ Parent | on HN ]