News for hackers. Stuff that matters.
Re:Benchmarking Opus 5 on SlopCodeBench by Espressosaurus on Monday July 27, 2026 @09:37PM (#49078185)
I’ve observed the degradation, but I suspect what’s happening is they’re tuning it for lower inference costs. Maybe turning down the amount of thinking, maybe quantizing, maybe something else.

It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.

Re:Benchmarking Opus 5 on SlopCodeBench by conception on Monday July 27, 2026 @10:22PM (#49078473)
https://marginlab.ai/trackers/claude-code/ their tracker generally shows that isn’t the case. The only times I’ve seen it drop is something broken and just before fable launched.
Re:Benchmarking Opus 5 on SlopCodeBench by nerdsniper on Tuesday July 28, 2026 @01:03AM (#49079578)
I mean they could just be routing known benchmark questions (which all of SWEBench are) to a full-performance variant.
Re:Benchmarking Opus 5 on SlopCodeBench by svnt on Tuesday July 28, 2026 @11:59AM (#49085865)
Is this using the api or using a subscription, though? The incentives are different for each, and it isn't the least bit unexpected that they would maintain API access quality while 'optimizing' the subscription experience to improve their margins (or losses)

It seems to do really this you would need to crowdsource it -- users individually give the lab access to a body of subscriptions normally used by average people, and the lab occasionally runs some masked version of the task through on diverse accounts.

Re:Benchmarking Opus 5 on SlopCodeBench by 8note on Tuesday July 28, 2026 @02:17PM (#49087829)
i thought i had noticed a degradation, but it turned out claude code had swapped itself back to opus.

might be the case for you as well