Re:Is Opus 5.5 nerfed? Thinking patterns changed on November 3
by kingstnap on Monday November 09, 2026 @01:54PM
(#50429871)
Lifenerf is really not sensitive enough.
Firstly, there is a lot of A/B testing. You would likely need at least 10 or 20 accounts to catch these random tests.
Secondly, the labs obviously run GPQA and MMLU like livenerf does and watch for their own regressions before doing A/B tests. These benchmarks are statistically the thing already being filtered out. You need tasks out of distribution. Ideally ones in distribution with the stuff they particularly frequently change (like safety or copywrite adjacent tasks).
These harnesses, and hidden system prompts, do get screwed with quite a lot in my experience. The models are actually remarkably consistent via API but there are tons of stealth system prompt and harness changes that lobotomize these models all the time.
Like despite being stochastic, try asking Luna to generate you a video game pitch via the API 100 times. It's almost always the same thing about lighthouses for some reason. That's actually why I tried to make a CLI entropy generator project.
Firstly, there is a lot of A/B testing. You would likely need at least 10 or 20 accounts to catch these random tests.
Secondly, the labs obviously run GPQA and MMLU like livenerf does and watch for their own regressions before doing A/B tests. These benchmarks are statistically the thing already being filtered out. You need tasks out of distribution. Ideally ones in distribution with the stuff they particularly frequently change (like safety or copywrite adjacent tasks).
These harnesses, and hidden system prompts, do get screwed with quite a lot in my experience. The models are actually remarkably consistent via API but there are tons of stealth system prompt and harness changes that lobotomize these models all the time.
Like despite being stochastic, try asking Luna to generate you a video game pitch via the API 100 times. It's almost always the same thing about lighthouses for some reason. That's actually why I tried to make a CLI entropy generator project.