hegstal

I've also seen similar recently at my place of work with LLM linting. Lots of hard to spot bugs in old very critical code have been caught. I remember seeing similar kinds of impacts back when linting or sanitizers were introduced. They're good for making it easy to encode more complex rules/areas-of-focus etc. that may have not been meaningfully possible with the more traditional tooling. Commercial LLMs in their current incarnations make for very expensive, but very effective intelligent linters.

On the other hand, my experience so far has been that they are terrible code-reviewers, to the point almost everyone ignores any kind of generic LLM review entirely (correctly - most of it is noise).

I feel the industry obsession with full-automation causes them to go down the wrong rabbit-holes on full-automation vs augmentation. In my view would be significantly more effective if the LLM review process in forge-tooling was designed with a human reviewer in the loop. The goal imo should be drastically improving the review efficacy & throughput of the human element, who needs to be acquiring confidence in & socializing the change in any sane org anyway. For example iteratively & interactively rubber-ducking out analysis + remediation, or interactively providing focusing guidance on the patch, instead of as agentic batch or harness processes as is cool nowadays. Sometimes a custom REPL is just better for an expert system. These things matter more as more and more code is LLM generated.

You could ask someone a math question, and they can write an equation down, and the equation can be mathematically sound, but it still not be the right form of the equation, or even relate to the question correctly at all. If all you're being asked to do is validate the mathematical soundness of the equation then sure, it's not that hard and it often mechanical, probably easier than having to generate the equation. But that isn't review -- that's cargo-culting what review is supposed to be.

To bring back to code: They are making the point that verifying "correctness" is not the be all and end all of code review. In every situation, there are multiple "correct" solutions, and only a few of them are actually correct and fit within the past, current and future context (technical and business). If someone isn't doing this, they aren't reviewing effectively and are acting as a glorified linter, which is an astonishingly low value-add use of human capacity.

There are situations where nobody really cares about the output, and nobody wants the true form of review, so it reverts to (at best) human-linting. In those situations, engineering capability of any kind functionally has no moat and all participants are equally fungible. If the business is dependent on engineering it's also a horrendous business. I would personally avoid being involved in these environments in an engineering capacity. I mean this in a diagnostic sense - it's a terrible career move, though there will always be busy work that's exactly what we have LLMs for.

Humans are fallible, so we review the code. They "hallucinate" and produce mistakes and go in the wrong direction. We evolved an entire discipline around processes, tools, reviews etc. to help remedy (not solve!) this.

For some reason, the moment those humans are being replaced by fallible, stochastic machines, we decide to just throw the baby out with the bathwater entirely and no longer leverage critical parts of this pipeline? Because it's inefficient and we're bottlenecking the process? The magic of LLMs has quite literally bought us more cycles to do this stuff, not less!

I use the things every day to an extreme degree and still I don't think I will ever quite understand the leaps in logic required to arrive at this position.

On the other hand, given there is prior art in Elfeed, why wouldn't it rapidly converge on feature parity?

One of the games they are also going after is Detroit: Become Human, and they have gone after things like GTA in the past. Just because they claim they are going after things for those reasons doesn't mean that's actually an accurate claim as to what they are trying to go after. Though it's good to point out who is actually (supposedly) responsible.

The anti-social 10x developer is more often than not the anti-social 1x developer that makes everyone else move at 0.1x or worse. Not that I directly blame them for it - there is far too much focus on the individual here. It's not so common they are bad people, but more that their own management doesn't provide them with the incentives to behave any other way. It's all about incentives. If the incentives are there, the parties involved will inevitably find a way to compromise in a way that works for them, but if the incentives are fundamentally misaligned, there is no way this gets resolved. The only option is to run.