there is also sentiment I see that engineers should move from the code review to the testing of the end result. which is nonsensical because with the code review you can catch problems which are borderline impossible to catch via testing. but

the code review requires a tamed agent producing comprehensible code.

I agree that we should be using these tools to work out more optimal solutions.

We need to go slow before we can go fast.

Code reviews matter in the sense that you need to check the code is reasonably sane occasionally, but that's not the same as doing PRs. Leaving checks to the end has always been a dubious strategy - it's a lot easier to catch problems if you're designing the code up front and then reviewing that things are on track during the dev process. Finding that there's a problem at the end is the worst possible time to change something.

AI has solidified notion that for me. When your team are cranking out PRs at a rate of several a day (not building faster; making small, atomic change requests of independent bits) you'll never have the bandwidth to review everything by hand. You need time at the start to understand the change, lots comms during the process of making the change, and lots of automation around deterministic tooling (linting, formatting, unit tests, integration tests, e2e tests, fuzzing, property testing, smoke tests, etc) to make sure your AI change hasn't broken anything. If you have that in place the need to review at the end of a change is greatly reduced. Down to zero in my opinion but your milage may vary.

I'm not reviewing entirely by hand - I'm using an LLM too - but I know enough to guide the LLM towards areas that look like problem areas.

Yeah, there was a lot of "things are working out so far" in that post. Perhaps they'll be proved right, but it set my spidey senses tingling.

> still adding unneeded complexity or choose the wrong abstractions in ways that will cause problems down the road, don't think about all the corner cases, and pick the wrong handling for error conditions.

These are all one of those things which have never been easily quantifiable and therefore not easily sold to the stakeholders (your boss, boss of your boss, etc.). The AI will only emphasize this condition.

Totally agree.

We decided that we know the system well enough that we can self approve PRs

I would say people might have slightly different opinions I. The company, but in general, if it is not critical and it can go to preprod to check, we can merge.

I particularly do an automated code review and use some setup to check idea, code and documentation. But if it is critical or too important to go wrong I seek for code reviews.

The cost of writing code has gone down but not the cost for engineering.

I see my friends struggling with vibe coded projects because they didnt plan how to grow the baseline. The line got blurred as it became cheaper to do-redo but you can check any vibe-coder code cost through time and it does increase at least linearly.

I will say that now it is easier to plan specific changes that can put in and out in case of problems whether before we would need to do several rollout steps so multiple people could work on a task.

Now several people work on several things and the struggles are others.

Code quality has been a pain for everyone. I think for an LLM doing things one way or the other is a matter of what fitted the chances by the prompt and not good practices or choices. By this I mean, if there are two ways of writing the same code and one is clearly harder to read or performs worse, it can be more a matter of being lucky/unlucky to get it right throughout the whole code.

There are weeks I get 30 PRs done, there are weeks I get 3 PRs, like it was before. But reading LLM code is painful. It is obtuse and without lots of guidance it invent rules to problems that do not exist.

Of course I see that from my bubble, but I guess as experienced engineers are seeing the problems piling up and it the models are getting more expensive and not getting better as faster as before. These three things can be where we will delimit the usage of LLM for coding.

I particularly like that I don't need to do some of the annoying tasks that took me only checking 48302 files to be sure I got all the same string correct because some past bad decisions. But when I am working in important stuff, the cracks in llm makes me much slower.

A personal challenge has been to keep up knowing the codebase well enough while people (including me) are dumping stuff in there.

I am also a principal engineer.

The fundamental issue is that traditional code review doesn’t scale.

I now use a “Controls” linter which applies deterministic methods to constrain AI output, requires explicit declaration when key resources are used (file system access, network egress, outside libraries, or whatever I feel is important), and ties directly to other external requirements for business level controls that may be audited. (And more, but this gets the idea across)

Individual control violations can block PR mergers, block release, or just flag for review.

I genuinely don’t care that much about AI code quality these days (though I exclusively use SOTA models like Astra, which could be related). I care about code structure and architectural quality. I’ve found those can be deterministically enforced and made visible. If structure and architecture are high quality, I have no need to nitpick code quality.

If I were working with humans, this system would be cruel. It is far harsher than any manual code review I’ve ever given to humans. I let humans get away with small issues all the time. Dealing with the consequences of small mistakes is how people learn. For anything that really matters, I am ruthless with AI developers.

The system itself needs to be audited, so code review is still necessary. It’s more spot checking than full review though. You need to be a decent traditional software developer and architect to build new controls and to run control audits.

I’d be really interested in how you do longing for architectural issues. I can’t see how that would work, especially deterministically?

I think we're in violent agreement that the sort of code review we used to do is no longer necessary, but that some sort of review still is.

However, "doesn't scale" doesn't sound like a valid reason to skip code review to me. More code does not directly translate to more shipped features.

I don’t completely disagree.

There are definitely areas where I’m reviewing individual lines. Protocols are high on the list. I always want to know that high-level services aren’t leaking their internal details to the outside world and that whatever entity owns a protocol is controlling what is being communicated.

AI is bad at certain kinds of abstraction (for now). For example:

Connect service A to service B. Service C exists in some deployments and will do the real work in those cases, but A must never use B as a passthrough to C or use C’s language. That would cause future planned changes in the app to require a massive refactor.

Then, later, in review: Why is A’s client protocol to B referencing functionality that only exists in C? B’s entire purpose in life is to prevent this!

I haven’t tried to solve this via automation yet. I think it will involve limiting what AI has access to.

Agree here. At my current place we have LLM apply deterministic rules where given thing cannot be checked statically. For example - all unit tests shouldn't access database. This can be checked statically. All comments should document why, rather than what - this cannot.

Humans tend to bring up things that matter to them + correctness, when they have time to think about it.

You can test comments with a decision model like Jev.

The context is: > At my current place we have LLM apply deterministic rules where given thing cannot be checked statically.

So in this example quality of comments is checked my LLM and not statically

I think it also depends on what kind of software you are writing.

If you're in finance, aviation, or some other industry where the cost of errors is very high, I do kinda want a human to look at every line of code even today.

Not only to look but also write

Generating? Fine. But type it out again when placing it into the codebase to make sure.

But I'm not worried, some planes still update with a floppy. I'm sure that code is handwritten.

How could I learn more about this linter approach (especially around architecture quality)? Any other terms to describe it or resources that explain it?

Sounds very interesting!

Declare and deterministically detect the entities you want to control. Figure out how to list them. Then, write rules that enforce structure and behavior of or between those entities.

It’s just lots of declaration, detection, and fancy tests. But you need to apply a lot of attention to detail and make sure you have confidence that a given rule is enforced, and that a given entity can be detected. There’s more to it than that, but if you start playing around I’m sure you’ll figure the rest out.

I will warn that the goal isn’t to get AI to write code like a human would. It’s to enable controlled, high-velocity AI-driven SDLC that scales to large, complex applications.

> both engineers and LLMs are still adding unneeded complexity or choose the wrong abstractions in ways that will cause problems down the road

Completely agree. But why do you think the reviewer would choose a better abstraction?

Yes, an experienced dev has a high chance of doing so. But in most teams there are not enough experienced devs.

And even I, an experienced dev, choose wrong abstractions all the time.

But fixing even large oopsies is just a prompt away. So imo it does matter less and less.

What was and is true: iteration speed just matters more. I would bet a lot of money that a team that goes from written code to merge in hours has a better quality codebase than one that takes weeks.

"Fixing large oopsies is just a prompt away" really depends on what you are doing.

I write software responsible for updating customer controlled devices. In theory, these devices may be in use for decades. I don't get to dictate when customers update their software. I have to use whatever software is on device as part of the update process, and the software update process really should work from every single version that's ever been deployed.

So whatever decisions I make / the team makes, generally we're stuck dealing with in some form for the device lifetime.