Re:Two things LLM coding agents are still bad at
by AIorNot on Thursday October 09, 2025 @12:25PM
(#45529870)
I think we need better code review tools in the age of LLMs - not just sticking another LLM to do a code review on top of the PR
Needs to clearly handle the large diffs they produce - anyone have any ideas
Re:Two things LLM coding agents are still bad at
by ngruhn on Thursday October 09, 2025 @02:50PM
(#45531575)
I was about to write my own tool for this but then I discovered:
git diff --color-moved=dimmed-zebra
That shows a lot of code that was properly moved/copied in gray (even if it's an insertion). So gray stuff exactly matches something that was there before. Can also be enabled by default in the git config.
Re:Two things LLM coding agents are still bad at
by VMG on Thursday October 09, 2025 @02:59PM
(#45531672)
Perfect. This is why I visit this website
Re:Two things LLM coding agents are still bad at
by erichocean on Thursday October 09, 2025 @03:34PM
(#45532088)
I used autochrome[0] for Clojure code to do this. (I also made some improvements to show added/removed comments, top-level form moves, and within-string/within-comment edits the way GitHub does.)
At first I didn't like the color scheme and replaced it with something prettier, but then I discovered it's actually nice to have it kinda ugly, makes it easier to detect the diffs.
Re:Two things LLM coding agents are still bad at
by karczex on Thursday October 09, 2025 @03:21PM
(#45531943)
Thanks:)
Re:Two things LLM coding agents are still bad at
by paulhebert on Thursday October 09, 2025 @05:36PM
(#45533364)
I would love if GitHub implemented this in their UI! There’s and issue: https://github.com/orgs/community/discussions/9632
Re:Two things LLM coding agents are still bad at
by godelski on Thursday October 09, 2025 @08:50PM
(#45534509)
That's a great solution and I'm adding it to my fallback. But also, people might be interested in diff-so-fancy[0]. I also like using batcat as a pager.
Re:Two things LLM coding agents are still bad at
by steveklabnik on Thursday October 09, 2025 @01:09PM
(#45530389)
I personally agree with you. I think that stacked diffs will be more important as a way of dealing with those larger diffs.
Re:Two things LLM coding agents are still bad at
by alfalfasprout on Thursday October 09, 2025 @04:28PM
(#45532688)
Yep, this pattern of LLMs reviewing LLMs is terrifying to me. It's literally the inmates running the asylum.
Re:Two things LLM coding agents are still bad at
by DrSiemer on Thursday October 09, 2025 @09:39PM
(#45534723)
When using a reasonably smart llm, code moves are usually fine, but you have to pay attention whenever uncommon words (like urls or numbers) are involved.
It kind of forces you to always put such data in external files, which is better for code organization anyway.
If it's not necessary for understanding the code, I'll usually even leave this data out entirely when passing the code over.
In Python code I often see Gemini add a second h to a random header file extension. It always feels like the llm is making sure that I'm still paying attention.
In particular, code review is one layer of the conventional swiss cheese model of preventing bugs, but code review becomes much less effective when suddenly the categories of errors to look out for change.
When I review a PR with large code moves, it was historically relatively safe to assume that a block of code was moved as-is (sadly only an assumption because GitHub still doesn't have indicators of duplicated/moved code like Phabricator had 10 years ago...), so I can focus my attention on higher level concerns, like does the new API design make sense? But if an LLM did the refactor, I need to scrutinize every character that was touched in the block of code that was "moved" because, as the parent commenter points out, that "moved" code may have actually been ingested, summarized, then rewritten from scratch based on that summary.
For this reason, I'm a big advocate of an "AI use" section in PR description templates; not because I care whether you used AI or not, but because some hints about where or how you used it will help me focus my efforts when reviewing your change, and tune the categories of errors I look out for.