This is possible, but not without being willing to change the level of abstraction at which you're working. We have 5 devs and changed from doing code review to plan review. You still peer-review what people are doing, but one layer of abstraction further back. Then when the PR gets raised there is a loop that just compares the implementation plan to the actual code and makes sure it was followed.
You still get to understand how things are getting solved i.e. "We're going to do this database migration in this way" but without reading the SQL. This is the plugin we wrote for it: https://github.com/until-dev/plugins
I hate reading agent-generated code and it's not a good use of my time, and I wanted a staistic to quantify by how much.My current estimate is $0.243 for a human to review one changed line and $0.114 in model spend for an agent to produce one benchmark-accepted changed line. That makes human review about 2.1 times as expensive under these assumptions.The agent calculation uses one public...
We ended up moving peer review to the implementation plan rather than the PR, then having a loop that validates the code against the plan when the PR is raised. That way the agent gets a CI failure if it deviates from the plan, which it then fixes or acknowledges. Anything with no differences gets merged without human review, differences get approved by the original person who peer reviewed the plan.
I puled some stats the other day for a presentation I'm working on about what we did:
Matched plan on first pass: 25% (169/677) Had differences: 75% (508/677)
Differences per PR: Median 2 Mean 2.60 P90 6 Max 20
1,757 findings:
- Missing (skipped planned work): 44.6% of findings, 51.7% of PRs - Changed (done differently): 42.7% of findings, 51.3% of PRs - Beyond (extra, still in scope): 9.1% of findings, 19.8% of PRs - Scope (unplanned feature): 3.6% of findings, 6.6% of PRs
I totally agree that with the right systems we can do away with code review, and it's a solidly impressive achievement by the model/harness. But three days an nearly $2,500 to fix something that is a very foreseeable requirement if you'd done a little bit of planning doesn't seem like a terribly impressive outcome.
I think the more interesting tradeoff here is that the system they had to fix feels like it was "vibed" rather than "engineered" (or maybe the lack of state tracking was a deliberate tradeoff, but it doesn't really read that way) it sounds like they're at the "find out" stage TBH. But maybe spending $2,500 to fix things that would have been easily solved by some forethought is a tradeoff worth making if you're shipping things at ludicrous speed. Maybe you just ship 10 features and only fix the ones that get traction and it works...?