News for hackers. Stuff that matters.
Re:AI usage generates a fragile codebase by stingraycharles on Tuesday November 10, 2026 @07:33AM (#50439428)
It’s also a great way for AI to keep creating backwards compatible bloat code when refactoring and whatnot because “tests are failing” when those tests are testing absolutely silly stuff (LLMs have a tendency to want to validate exact output rather than behavior).

I don’t think anyone has figured out the proper solution yet, I personally try to get the LLM to write code similar as I would have written it myself.

Re:AI usage generates a fragile codebase by drooby on Tuesday November 10, 2026 @07:47AM (#50439597)
What do you mean "exact output rather than behavior"?

If you mean asserting against hardcoded literals, then that is the preferred testing shape in most cases. And for good reason.

If you change production code and a unit test fails, that is useful. Idk why that's silly. If you want to test more complex interaction write a UITest or integration test.

I honestly think we're all living in different worlds. There's probably large differences in capability on a per-language, per-team, per-codebase, per-engineer environment that are producing wildly different perceptions in output quality.

Re:AI usage generates a fragile codebase by stingraycharles on Tuesday November 10, 2026 @08:08AM (#50439784)
It would add tests for exact error messages, for example. It would add tests that the process exited with very specific error codes. Basically tests that I would never ever write myself.

And yes everyone works in different domains. I work at a database vendor myself, so mostly lower-level work.

I’m very happy and successful with LLMs, I just don’t think that these “just move all your tests to your type system!” or “get 100% test coverage” rules are an answer in the long term as we haven’t yet figured out how to do this well.

I personally treat tests as components that need to be well architected, maintained, and add them risks increasing technical debt and needs to be treated as such.

Re:AI usage generates a fragile codebase by drooby on Tuesday November 10, 2026 @08:46AM (#50440279)
That feels like a pretty uncharitable interpretation of my original comment.

I was describing general strategy, that when applied in concert is more than the sum of its parts.

E.g I mentioned property-based testing, that is another angle at which a test avoids assertions that are too fine grained.

Re:AI usage generates a fragile codebase by stingraycharles on Tuesday November 10, 2026 @09:49AM (#50441176)
Yeah I just don’t consider adding more or better tests as a solution to fragile code, which is the topic of the conversation.

I think it has much more to do with overall architecture and design, and as for now, human oversight.