News for hackers. Stuff that matters.
Re:Claude Opus 4.7 by verdverm on Thursday April 16, 2026 @11:08AM (#47794302)
Some of the benchmarks went down, has that happened before?
Re:Claude Opus 4.7 by andy12_ on Thursday April 16, 2026 @11:24AM (#47794599)
If you mean for Anthropic in particular, I don't think so. But it's not the first time a major AI lab publishes an incremental update of a model that is worse at some benchmarks. I remember that a particular update of Gemini 2.5 Pro improved results in LiveCodeBench but scored lower overall in most benchmarks.

https://news.ycombinator.com/item?id=43906555

Re:Claude Opus 4.7 by grandinquistor on Thursday April 16, 2026 @12:05PM (#47795434)
looking at the system card for opus 4.7 the MCRC benchmark used for long context tasks dropped significantly from 78% to 32%

I wonder what caused such a large regression in this benchmark

Re:Claude Opus 4.7 by grandinquistor on Thursday April 16, 2026 @11:14AM (#47794415)
Probably deprioritizing other areas to focus on swe capabilities since I reckon most of their revenue is from enterprise coding usage.
Re:Claude Opus 4.7 by cmrdporcupine on Thursday April 16, 2026 @11:27AM (#47794669)
It's frankly becoming difficult for me to imagine what the next level of coding excellence looks like though.

By which I mean, I don't find these latest models really have huge cognitive gaps. There's few problems I throw at them that they can't solve.

And it feels to me like the gap now isn't model performance, it's the agenetic harnesses they're running in.

Re:Claude Opus 4.7 by nothinkjustai on Thursday April 16, 2026 @11:58AM (#47795306)
Ask it to create an iOS app which natively runs Gemma via Litert-lm.

It’s incredibly trivial to find stuff outside their capabilities. In fact most stuff I want AI to do it just can’t, and the stuff it can isn’t interesting to me.

Re:Claude Opus 4.7 by ACCount37 on Thursday April 16, 2026 @11:19AM (#47794502)
Constantly. Minor revisions can easily "wobble" on benchmarks that the training didn't explicitly push them for.

Whether it's genuine loss of capability or just measurement noise is typically unclear.