Specialized AI code review software is so pointless though. Back when agents were dumb about git surgery and tool use it might have had a purpose, but now you could replace coderabbit with a skill and I bet the results would be better in some cases.

Mechanical code review by something like sonar qube is much cheaper. Use that for low bar quality gate and AI after.

AI code review is startlingly effective. Continually finding things me and my colleagues never would. Well the decent models do. Maybe not so much the cheap ones.

I’m not saying that it’s not useful, I’m saying that it’s not useful in a “human in the loop” situation. This type of AI-to-AI review should be done agent-to-agent, not through Github PRs with tools like Devin.

In a manual review, I then expect all “machinery” to already be properly reviewed, and can focus on design / architecture. I would like an AI assisted review tool to make that part easier, not do the actual review for me.

I’m not convinced by the automatic agent-to-agent thing. I find that, if I manually ask a standard harness “Review a..b, individually and for combined effect”, I get some mix of catching genuine errors (some quite deep), incorrect flags where the correct course of action is to ignore them or modify the commit messages, and comments where the correct course of action is to think deeply.

If I were to automate the back and forth, I would get spurious changes that “fix” what wasn’t broken and a removal of the actual interesting bits.

this touches one of the reasons opus 5 came by default with 200k context and 1M gated behind usage tokens, auto-compaction by default, and it keeps telling you to clear and start from scratch all the time, AI is does a better job fresh, telling multiple agents to pipe around ideas without the human intent is the worse thing you can do