News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jiggawatts on Wednesday January 01, 2025 @03:54PM (#42569204)
A real acid test is to use numbers that confuse LLMs such as:

“Which is heavier: a bag with 9.9 pounds of fluff or a bag with 9.11 pounds of steel block?”

Try swapping the numbers and the materials, run each test at least 10x times.

There are no comments on this comment yet.