News for hackers. Stuff that matters.
Re:2025: The Year in LLMs by techpression on Thursday January 01, 2026 @03:56AM (#46452448)
Just last week Opus 4.5 decided that the way to fix a test was to change the code so that everything else but the test broke.

When people say ”fix stuff” I always wonder if it actually means fix, or just make it look like it works (which is extremely common in software, LLM or not).

Re:2025: The Year in LLMs by simonw on Thursday January 01, 2026 @04:01AM (#46452474)
What did Opus do when you told it that it shouldn't have done that?
Re:2025: The Year in LLMs by layer8 on Thursday January 01, 2026 @05:25PM (#46458801)
It apologized. ;)
Re:2025: The Year in LLMs by viraptor on Thursday January 01, 2026 @04:21AM (#46452582)
Sure, I get an occasional bad result from Opus - then I revert and try again, or ask it for a fix. Even with a couple of restarts, it's going to be faster than me on average. (And that's ignoring the situations where I have to restart myself)

Basically, you're saying it's not perfect. I don't think anyone is claiming otherwise.

Re:2025: The Year in LLMs by b3kart on Thursday January 01, 2026 @04:37AM (#46452668)
The problem is it’s imperfect in very unpredictable ways. Meaning you always need to keep it on a short leash for anything serious, which puts a limit on the productivity boost. And that’s fine, but does this match the level of investment and expectations?
Re:2025: The Year in LLMs by techpression on Thursday January 01, 2026 @06:24AM (#46453241)
It’s not about being perfect, it’s about not being as great as the marketing, and many proponents, claim.

The issue is that there’s no common definition of ”fixed”. ”Make it run no matter what” is a more apt description in my experience, which works to a point but then becomes very painful.

Re:2025: The Year in LLMs by baq on Thursday January 01, 2026 @06:04AM (#46453131)
Nice. Did it realize the mistake and corrected it?
Re:2025: The Year in LLMs by techpression on Thursday January 01, 2026 @06:17AM (#46453202)
Nope, I did get a lot of fancy markdown with emojis though so I guess that was a nice tradeoff.

In general, even with access to the entire code base (which is very small), I find the inherent need in the models to satisfy the prompter to be their biggest flaw since it tends to constantly lead down this path. I often have to correct over convoluted SQL too because my problems are simple and the training data seems to favor extremely advanced operations.