I would assume though advanced reasoning models would be best, especially since it's out of band. If I were building it maybe I'd run two adversarial models.
It leaves comments though like a human reviewer would. I assume the key is a good prompt making sure the comments aren't too frivolous.
But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.
LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.
Is that hotel example real? Curious how exactly you employ this technique—my naive idea was, if talking software development, a sort of 'sanity-check auto-linter agent' catch errors on a regular basis (every 10 seconds, every write, w/e).
I refactored the tool out of my application and it's available now at https://endpointevaluator.com . There's nothing there someone couldn't build themselves, but then you have to spend that time building and maintaining it. I think the ongoing and long term cost of maintaining all this LLM-generated software is under-appreciated. So hopefully there is still space for outsourcing tools that are generic enough to be used by many and provide good value for cost.
The instinct seems good because it's impossible to keep up with all the details if you are running AI full-blast. Absolutely impossible. So testing outputs makes sense.
I have a hard time seeing exactly how we get from here to there. But intuitively I would not be surprised. One of these thing where quality may drop 20% but you can scale 100x.
Even with the same model and the same input, the output is inconsistent. And what I've observed is that as the size of the input and output grows, the consistency and accuracy of the output seems to decrease. It gets more complicated when you don't control the full input, such as a chatbot with customers.
I think the problem remains even if it can be mitigated by freezing the model and the hardware, which carries the tradeoff of requiring a model you can download and run on your own so you can't use the SOTA models.
What exhausts and leads to burnout, is probability.
Sometimes you get so much dopamine if it goes right.
And sometimes you feel like an idiot because LLM does so many mistakes even if you think that you prompt it right.
As the pace is so fast, you change your mind’s state so frequently and quickly that it may lead to a (mild) burnout.
This is so true. Cognitive brute-force always when dealing with AI.
Or maybe you can just get too spoiled with it grokking your intent, then become so vague that your vague ideas are actually just bad ideas. Certainly has happened to me.
Hasn't everyone already got agents directly adding themselves to PRs and leaving comments (occasionally useful)?
I personally would never dump a claude code response into a PR body an ask someone to address it. I can't believe any developer would find this to be appropriate or fulfilling the duty of reviewing code.
I think it's pretty clear that what you should do instead is go over the result and communicate to the PR author anything you think should be addressed. You don't just say "here are 8 things you might want to address but I cant actually speak to any of them."
To add to this, the fact that Claude so often tries to deviate from defined architecture can be really frustrating if you're working in a mature codebase.
Unfortunately, the results speak for themselves and I could probably find dozens of edge cases which the AI code review caught which would have otherwise been missed, probably until production.
To be absolutely clear, AI should be used for PR review. It should be used many times. By the PR author and by all the reviewers. It should not just be piped directly from the agent to the author inside the PR. That causes the author to triage every comment.
If the AI can review the pr, it can do so before the PR exists in the first place.
It's a tool/skill/prompt. Just share it with the end user or encode it in the codebase in some files.
It may actually be even more useful as that person will use it often and improve it or can it.
The exact same skill in the hands of one person will have vastly different outcomes in the hands of a different person. The review skill I built myself has been shared with folks. They (say they) use it before they put up a PR. I still catch things with the same skill. That is evolving as I catch the model(s) at BSing. Even if I would re-share it all the time, when I catch it, these other people using the same skill wouldn't catch it.
And that is exactly why building a general purpose review agent (or skill) is harder than me having my own evolving skill. If I have to "properly" release a skill/some automation, I will likely err on the side of not having too many false positives. That is harder than still relying on a human to review the AI reviewer. The same people that were really good actual human reviewers of a PR are probably the ones that are good doing the same thing while AI assisted. And the ones that really needed these other human reviewers and processes to help them be productive are probably going to produce a vibe coded mess if left to their own devices with nothing but some AI.
I'd guess people truly are finding things pre-review. It's just that LLM review seems to have a limit to the number of problems it can or will find. While at the same time, LLM-written code can be an almost limitless source of bugs and errors.
It has no sense of when an architecture is doomed and needs fundamental changes. Instead, it will happily continue to chase individual bugs nearly to infinity.
But, maybe your review skill is better than the ones I've tried.
A well authored CI review process is significantly better than any human could do. We have the AI review not only the changes but clone and investigate all related repositories that integrate with the code to evaluate interactions, check all open and historical issues that may be related, review production/development logs for the feature, etc.
It will then also go update issues/tasks to reflect the changes after the merge.
Similarly you can have automated agents watching post-deployment to mitigate risk in the "a new feature just went out" time-frame. Just having someone check the related logs in the hours after deployment can catch a lot of issues that otherwise would turn into a fire-drill.
The human code monkey sometimes thinks too highly of the task of coding. That's never once been the job to be done.
Every engineer is going to become a master of understanding and measuring need, opportunity cost, organizational support and headwinds, political will, etc. These are the true features of the job. The coding part will fall away entirely.
Deliver value. Code used to be the expensive part, but now it's becoming cheap, fungible, and ephemeral.
...of course you're likely just trolling, hard to say with the drivel coming out of Silicon Valley the last few years. In that case: my apologies and keep up the good work :D
The only answer I can imagine is that it supports reviewing patches from people who aren't using AI or review by humans but parent doesn't mention this.
> you should have 2+ developers looking at most PRs
It’d be nice, but usually not the case in my experience. More eyes is better. AI review should not replace human review, it should supplement it. I find myself spending more time doing end-user testing instead of looking at code vs before.
works perfect
https://github.com/dzmitry-lahoda/dz/tree/main/agents/skills...
burns half of day sub of astra for 200 USD. runs 1 hour on our repo.
finds bugs missed by coderabitai, devin-integration-bot, codex and copilot (I ask them first until nothing found, yet my orchestration finds more).
I do not hardcode our application in orchestration, but run subagent for applicationdomainproduct detection.
also I use agy 200usd sub for second-opinion as one of steps for false positive elimination.
I would if the PR was clearly written by AI. I'm fine with the PR author using AI, but only to draft the PR. They should be editing the shit out of it for the final version before submitting it.
- Human -> AI: OK
- AI -> AI: OK
- AI -> Human: Not OK (at least here)
AI code reviews are the same as AI pull requests. Do you want that firehose? The suggestions might be good. But do you want to add a deluge of work items to that part of the workflow? Do you want the PR owner to be the one to triage feedback before it gets to them? After they have already done it themselves, perhaps with an even better model?
of course not, it's disrespectful to the author even if they used an LLM to generate the code. what you should do is actually try to understand what the LLM is saying about the proposed changes, check whether it's talking shit or legit, and if legit, rephrase in your own words why you think a certain thing should be changed
The PR author asks for feedback if it needs feedback, otherwise it merges it, period.
I don't know why and when the world got convinced that all this bureaucracy is a "best practice", when it's just a practice, that can be good, or a waste of time depending on many factors.
Like do doctors when analyzing your medical records are like "I have a diagnosis, but first let me have two colleagues review it"? No, unless they are at the beginning and it's part of the process or they need one.
And that's way more serious than the crap most HN is writing.
Maybe yours was, albeit it sounds strange because hourly it doesn't even happen in intensive maternity-unity cares where there's a high risk for the child's health.
The author does not triage the feedback. Review bot assesses priority and agents fix the issues the automated review discovers. Nothing is blocked by the bot, humans can ignore if they think the feedback isn't helpful.
We use AI heavily in development but everyone has their own setup and way of approaching use. AI in PRs provides a consistent review layer beyond what the engineers do themselves, and catches a previously undiscovered issue in about 75% of the PRs.
kind of high level.
did you noticed that astra started to write better comments which look as it understands something?
You should already have 2+ developers looking at most PRs. And these developers should absolutely use AI. The PR author should use AI.
But what you should not do is pipe the AI output directly into the PR and tell the PR author to deal with it. That's adding noise to the PR review process. Everything it says is something the PR author needs to validate as relevant, helpful, etc. A human needs to do that before confronting the author with it.
You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?