Re:Ask HN: How to deal with shipping AI-written code you don...
by edwin0 on Monday October 19, 2026 @08:36AM
(#50151648)
The part that makes me uneasy is when the AI writes both the code and the tests. They might agree with each other and still miss the same thing. How are people here checking for that without doing the whole job twice?
[ Parent
| on HN ]
Re:Ask HN: How to deal with shipping AI-written code you don...
by webarchitect on Monday October 19, 2026 @11:54AM
(#50153955)
I like fault injection (fuzzing, chaos), and mutation testing. I think introducing these helps the AI write better tests and helps having some confidence in the tests that are written.
Re:Ask HN: How to deal with shipping AI-written code you don...
by ch_sm on Monday October 19, 2026 @11:11AM
(#50153438)
Not sure about that. This was structurally the same before AI: the author of the code wrote the tests. The tests still helped.
Re:Ask HN: How to deal with shipping AI-written code you don...
by seanw444 on Monday October 19, 2026 @03:17PM
(#50156715)
The author of the code was physically touching the internals and potentially intuiting obvious shortcomings as they worked on it. That very crucial part is now gone.
Re:Ask HN: How to deal with shipping AI-written code you don...
by fractallyte on Monday October 19, 2026 @12:58PM
(#50154810)
You should be using two (or even three) different LLMs. For example: Claude to do the coding, and Codex to do an adversarial review. This can go back and forth for several iterations before converging on an acceptably "correct" version of the code...
Occasionally, Claude may not be able to fix the issue – and it may turn out that Codex can, because it has a different approach.
Either way, at least two LLMs is the only way to get (mostly) reliable code.
Re:Ask HN: How to deal with shipping AI-written code you don...
by eivindmeyer on Monday October 19, 2026 @02:08PM
(#50155785)
You can automate this entire orchestration with zeroshot.sh, mentioning it since it is relevant here.
Re:Ask HN: How to deal with shipping AI-written code you don...
by NishanStepak on Monday October 19, 2026 @01:45PM
(#50155461)
Claude, ChatGPT, Google Gemini, then Replit...
Re:Ask HN: How to deal with shipping AI-written code you don...
by Bootstrapper909 on Monday October 19, 2026 @06:38PM
(#50158987)
We split them. The agent writing the code never writes the checks it's judged by, and those separate checks caught way more than its own tests did.
Re:Ask HN: How to deal with shipping AI-written code you don...
by whysosuresh on Tuesday October 20, 2026 @07:20AM
(#50164571)
Yes, this is exactly the problem that used to worry me. Most of my current repo is written by Claude, and I need to have a method to tests the tests.
I've started using Stryker. We have a CI job that mutates the code before running the tests. In the first run, it has returned hundreds of code changes confirming that the tests are biased. I haven't yet fixed those issues, but for now i am banking on this approach.