News for hackers. Stuff that matters.

A comment on Claude Opus 5.5 Intelligence, Performance and Price Analysis (Max). Parent comment

Re:Claude Opus 5.5 Intelligence, Performance and Price Analy... by tedsanders on Tuesday September 22, 2026 @07:09PM (#49809554)
How many tasks were in this dataset?

If it's a single task where it dropped from 50 to 25, it could be random variation (not saying it is, but it could be). If it's the mean over hundreds of tasks, that suggests a problem with either the eval code/harness or our API.

[ Parent | on HN ]
Re:Claude Opus 5.5 Intelligence, Performance and Price Analy... by breckenedge on Thursday September 24, 2026 @08:09PM (#49838492)
Just about 90 PRs in that dataset to review.
[ Parent | on HN ]
Re:Claude Opus 5.5 Intelligence, Performance and Price Analy... by tedsanders on Friday September 25, 2026 @06:01PM (#49850511)
Is it 90 independent tasks, or a single task with 90 pieces?

(it matters if they are independent or dependent)

[ Parent | on HN ]