News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zahlman on Wednesday January 01, 2025 @04:54PM (#42569627)
We aren't testing whether the model's results are stable or correct for a given class of problem. The goal is to establish whether the model can reason.

Nothing capable of reasoning would contradict itself so blatantly and in such a short span while failing to indicate any kind of uncertainty.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by Isinlor on Wednesday January 01, 2025 @05:37PM (#42569892)
Reasoning is not a binary skill.

And failure modes of other types of reasoners do not need to be the same as the failure modes of humans.