How to control unit tests in Agentic Pipelines
Have some wins but many struggles here
When generating new features using agents, we must then update the unit tests
This is counter to the purpose of unit tests.
Allowing the system to update its unit tests is introducing risk and uncertainty, even with scoped PRs
Not allowing the system to update unit tests will negatively impact velocity…
How are folks in the field handling this? Have a real rube-Goldberg situation going on here
(
2 comments
| 2 points
| on HN
)
The Rube Goldberg trap usually happens when a single agent has write access to both src/ and existing tests/ in the same pass. When a test fails, an unconstrained agent takes the path of least resistance: mutating the assertion to match its broken code rather than fixing the logic.
Teams getting velocity without sacrificing trust are separating the concerns along three lines: Bullet Split new vs. existing tests via file permissions: Let the coding agent freely create new test files for net-new behavior, but make existing test files read-only during feature implementation. If an existing test breaks, the agent must either fix its implementation or stop and flag why the old requirement is obsolete.
- Flip the order (Agentic TDD): Have a spec/test agent translate the ticket into failing tests first. Pause for a quick 2-minute human review of just the test diff—reviewing assertions is much faster than reviewing implementation—then lock the test files and let the coding agent iterate until the suite passes.
- Test behaviors, not mocks: If every new feature breaks dozens of existing unit tests, the suite is likely coupled to implementation details (heavy mocking, internal method calls) rather than public interfaces and outcomes. Moving toward black-box integration or contract tests makes existing suites resilient to new features while catching actual regressions.
In practice, humans own the invariants (approving changes to existing assertions) while agents own the implementation and net-new test boilerplate. You trade a tiny bit of raw generation speed for eliminating the slow, painful PR reviews where nobody trusts the green checkmark.
AI now can do very accurate and trustworthy e2e tests, and this is how your end users will use the product anyway, but you can run a lot of scenarios in parallel and fix bugs faster than before, the approach of testing need to change imo