Re:How to control unit tests in Agentic Pipelines
by n4pw01f on Tuesday November 10, 2026 @02:24AM
(#50437151)
Here’s what we got -> we already have an iterator rolling out invariants, but maybe this is the way:
The Rube Goldberg trap usually happens when a single agent has write access to both src/ and existing tests/ in the same pass. When a test fails, an unconstrained agent takes the path of least resistance: mutating the assertion to match its broken code rather than fixing the logic.
Teams getting velocity without sacrificing trust are separating the concerns along three lines:
Bullet Split new vs. existing tests via file permissions: Let the coding agent freely create new test files for net-new behavior, but make existing test files read-only during feature implementation. If an existing test breaks, the agent must either fix its implementation or stop and flag why the old requirement is obsolete.
- Flip the order (Agentic TDD): Have a spec/test agent translate the ticket into failing tests first. Pause for a quick 2-minute human review of just the test diff—reviewing assertions is much faster than reviewing implementation—then lock the test files and let the coding agent iterate until the suite passes.
- Test behaviors, not mocks: If every new feature breaks dozens of existing unit tests, the suite is likely coupled to implementation details (heavy mocking, internal method calls) rather than public interfaces and outcomes. Moving toward black-box integration or contract tests makes existing suites resilient to new features while catching actual regressions.
In practice, humans own the invariants (approving changes to existing assertions) while agents own the implementation and net-new test boilerplate. You trade a tiny bit of raw generation speed for eliminating the slow, painful PR reviews where nobody trusts the green checkmark.
The Rube Goldberg trap usually happens when a single agent has write access to both src/ and existing tests/ in the same pass. When a test fails, an unconstrained agent takes the path of least resistance: mutating the assertion to match its broken code rather than fixing the logic.
Teams getting velocity without sacrificing trust are separating the concerns along three lines: Bullet Split new vs. existing tests via file permissions: Let the coding agent freely create new test files for net-new behavior, but make existing test files read-only during feature implementation. If an existing test breaks, the agent must either fix its implementation or stop and flag why the old requirement is obsolete.
- Flip the order (Agentic TDD): Have a spec/test agent translate the ticket into failing tests first. Pause for a quick 2-minute human review of just the test diff—reviewing assertions is much faster than reviewing implementation—then lock the test files and let the coding agent iterate until the suite passes.
- Test behaviors, not mocks: If every new feature breaks dozens of existing unit tests, the suite is likely coupled to implementation details (heavy mocking, internal method calls) rather than public interfaces and outcomes. Moving toward black-box integration or contract tests makes existing suites resilient to new features while catching actual regressions.
In practice, humans own the invariants (approving changes to existing assertions) while agents own the implementation and net-new test boilerplate. You trade a tiny bit of raw generation speed for eliminating the slow, painful PR reviews where nobody trusts the green checkmark.