Code review is great if the code isn't a surprise, the architecture was already agreed on, the AI reviewers are already happy, and you just need a final pass.
Code review was already terrible before AI if you were slopping down weeks of work on a poor teammate to sift through.
I tell the younger engineers at my company the same thing today I said 5 and 10 years ago. Make the review easy for another engineer to do it, and they'll do it. Make it hard on them, and you won't ship shit.
But the "make review easy" part is one where I'm failing (a bit) to keep up, I feel.
Also, what does "a final pass" look like to you? E.g. if a PR is 3000 lines... I'd guess one approach is enforce smaller PRs. I've opened a few, looked at the line count, and closed them, to be redone/broken into stacked PRs.
If I'm reviewing a PR that I'm familiar with the plan/work on, my "a final pass" is often just "read and suggest touch-ups of the docs". But that's really just the first step of a "full review" too, so it skips other stuff, and I know that isn't a proper review in the back of my head, but on to the next thing! :|
(Because of several factors including joining a much less mature org git-wise this year, I'm full of internal questions/dilemmas around PRs that I'm working on feasible solves for)
At least for me. Am I missing something?
Also mind: one doesn't only have to review changes code, but also review what hasn't been changed (are preconditions of other code still met?)
For me, what works is highly templated directory structure and rules that linters can quickly check and enforce. Every PR is a cookie cutter design filled in. There are only a handful of things you need and they’re all just CRUD at the end of the day. You can employ the same very few design patterns and primitives across a lot of your stack. Review is so fast, if you trust your CI to do its job and the developer to actually QA their own work.
What isn’t working is the sheer volume of PRs. Even with CODEOWNERS directing attention to only high impact changes, it is a constant distraction. I’m not sure how to handle SOC2 where we need a human reviewer on all code. Lastly, agents shit the bed when they touch spaghetti, esp when it’s sitting next to structured patterns. IME, the best way to get around this is to just rewrite features from scratch. But in this current world of instant completion and reward, it’s been challenging to convince people to commit to more ambitious lifts.
Finally, I want review more than ever. A team can’t keep up with the pace of change even if it was all perfect. I like the idea of more up front planning, but the code is ultimately what matters.
This all has changed. I’ve only seen gaslighting (in the form of “of course we expect you to read the code LLMs is generating”), lying along the same lines, or the realistic people who acknowledge they don’t read any of it. The former camp is mostly traditional engineers, and the later is mostly new or traditional non-engineering roles like recent product management coding converts. I think it would help everyone if we just normalize the fact that no one if reading code much anymore. It’s just not feasible. Write a bunch of creative reviewing bots or SKILLs and just leave it at that. It’s not a great answer, but like I said I haven’t seen any alternatives.
> if you trust your CI to do its job and the developer to actually QA their own work.
I trust the CI, but the CI is doing what the CI is written to do. I don’t know how you can achieve the second part in a large team/corporate setting or an open source setting.
Yes! I think a lot of struggle comes from not having the same reliance on LLMs across the team. If developer A is not opening IDE and not reading the code to do work and developer B does not use LLMs, types code letter by letter and reads every text they encounter - it will not go well for the team. I also think many devs rely on LLMs more than they're willing to admit to the management and the rest of the team.
1. Code review is now less about mechanical validation (which often happens in linting, AGENTS.md, etc.) and more about education and understanding what's changing 2. Engineers need a tool which allows them to focus their attention on parts of changes, without having to review everything all at once
Would love to hear what you think!
Working on autonomous development with agents and no human code review. The future is human oversight, without humans in the loop. A PR centric cycle is like being in a factory and standing next to a conveyor belt of PRs. Humans can't possibly review properly. They have to take a few steps back and start looking at the entire machine, evaluating the system as a whole.
Then something is either wrong with your conception of PRs or whatever process you have been submitted to. PR is asking for feedback from people that already understand the software. It's also a learning place for anyone that is not yet familiar with the part being modified. So you have the author, the reviewers, and the learners. Which is three separate sets of people.
The PR review itself shouldn't take a lot of time (10mn is already a lot), because it's either a small change which is easily understood[0] or it's a more complex one [1] that should have a design discussion thread attached. This discussion is orthogonal to the code itself. Both should have a description good enough for the reviewer to have an idea why the change is necessary before looking at the code.
So a PR is an exchange of ideas around a real artifact, the code. If no idea is being exchanged, then it's merely a gatekeeping activity.
[0] https://cgit.freebsd.org/src/commit/?id=73b1453e6d87f9af4c47...
[1] https://cgit.freebsd.org/src/commit/?id=49a3cbc47f8e721a787e...
1. Hunk checkpoints - If it's a moderately sized PR (which most of the agent PR nowadays are), I as a developer find myself like to think in terms of hunks whether I finished reviewing that part or not. GitHub only allows you to mark only files as reviewed and nothing else.
2. I am trying to add semantic diffs to code reviews. That way a developer can actually understand the changes in the structure of the code instead of relying on reading plain text changes only. For ex: a function rename can simply compress 20 lines of changes into a single line.
3. Any agentic reviews and/or automated deterministic reviews can use both of the above to mark some simple things as reviewed in order to help the reviewer more.
4. This is in addition to existing review features from GitHub and the interdiff feature from other review tools.
A small mockup is available on the marketing website at https://juju.bi
I appreciate any comments on this approach. What do you guys think?
How long are your files that you can't easily "finish" a file?
(Semi serious here.)
I’ve played with some bookmarklets that modify the GitHub PR ui and attempt highlight these situations, but my setup is brittle (just trying to identify basic patterns in the text diff, ex “these are [oldname] replaced with [newname]”, which is not necessarily the same thing as an IDE rename-this-function action). I’ve been wondering if the answer here is in doing a diff on a compiled intermediate form of the program, visualized in some way - this would have to be language-specific though, I guess, and is well outside my expertise.
Separate from the diff aspect of reviews, I would also appreciate some more control in a PR tool over how to layout a review (what files/hunks to be shown, in what order). My team attempts to do this with carefully organized commits, so that reviewers step through them one by one, but the GitHub UI leaves a lot to be desired here (files appear in a static order top to bottom; comments and discussions can’t be stuck to one commit and show up in other commits; general bugs in the web view). I would appreciate being able to control how others see my PR if there’s room for that in your product.
Anyways, I joined your waitlist and am curious to see what you come up with - I think there are improvements to that can be made to code reviewing (that don’t involve AI - everyone is already putting AI review in their PRs), and am willing to try anything that takes a swing at that.
I am quite interested in knowing more about this. Would you mind give me a small example? Do you want to control individual hunks order or just files?
As for changing the display of hunks within one file, I think that could be interesting for a couple reasons (but I totally get that could be obnoxious to both build and present a UI for). I can try to dig up an example later, but off the top of my head there's two situations I've seen infrequently this might help with? (I'm sorry if I'm vague - I cannot share any of the code I do for work, which is 99% of the PRs I deal with).
1. A specific file was refactored in such a way that code was moved, deleted, and added, and the interpreted hunks together are intermingled. I feel like I see diffs where it's hard to tell what's going on because there are delete/addition/delete/addition blocks alternating, but they form separate contexts/actions (and often have different indentation levels or other styling quirks) it's just the diff algorithm displaying them poorly that is causing the problem. This is normally the spot where I might switch between a side-by-side diff viewer and a unified viewer, but sometimes neither is good. (This could probably be solved with more careful individual commits, although sometimes this is unavoidable).
2. We have a few large modules (singluar classes in 2000+ln files) that often have a wide public interface, with multiple functions eventually calling the same internal helper (a trivial example would be a multiple separate search functions providing different kinds of windows/orderings/etc of a data source, sharing an private function that really does most of the main work), but this private function is not first in the file. It'd be helpful to order this to be seen first - "here's what I changed' followed by 'here's how it is used now'.
Maybe what I want could just be solved with a commit-by-commit ordering of things, but just showing multiple commits together on screen, with later commits coming further down the page (as opposed to showing the whole unified diff of a group of commits - Github does that, and that is pretty much never what I want other than 'what's changed since I last saw this').
I can attempt to find some specific examples later this week and try to explain these situations clearer if the above is confusing.
We need some sort of summary mechanism or invariant prerequisites to keep track of with tools. When contributions start violating them, we might need a PR to update the architectural invariant or roll them back. This is a decision when a person needs to have a look and act.
35 PRs on my client's codebase would be no big deal because I took a lot of time to understand the fundamentals of the business around it. I can recognize within seconds if a PR is an easy merge or a 5 minute phone call.
If you are struggling with PR reviews, this is a sign that something is wrong with the coal mine, not with your birds.
I'm glad you can.
Even before agents, I'm pulled to work on at least a couple different projects a year, the successful ones with teams of 10s of people working on them. If I stay longer than a couple of months, I eventually become the expert on some of the things I did, so I get pulled years later to review something I haven't touched in ages, which has changed requirements and grown a bunch of dependents.
That was the reality of working on a team with hundreds of people and dozens of related products.
Now agents are “helping” people increase complexity and velocity 10-fold.
You can call it a problem in the coal mine, but there's no going back. GitHub simply has no choice but to handle this new scale.
I could, before LLM-authored PRs. They broke all my heuristics for what to check. LLMs can write code that looks incredibly professional and well thought out, and nowadays they almost never make the usual human mistakes (like off-by-one errors). But at the same time, they can make incredibly silly mistakes of the kind people would never make (like inventing brand-new parameters for some remote API call).
It feels like everything I’ve learned over the years about reviewing code no longer applies.
There’s SO much to be done around Code Review, I’m very surprised GitHub isn’t doing more.
They have the reach already, they have years of real review data. And yet, the PR itself is not using any of the newly available methods for more efficient review.
Agents are hammering PRs out left and right, and yet here I am still staring at the raw diff of 35 PRs and end up being called the bottleneck.