It’s not a reason for the wide chasm though.

If you told me that some significant number of Ant findings were duds, then I could reconcile that with what I saw first hand.

But TFA is making a more extreme statement

Likely difference between individual experience and group experience.

Useful to figure out if there are other cases like yours, so then commonalities can be inferred.

Totally

I’m just sharing my data point

The reason is that's just how AI is. It has a very jagged edge of experience. In some ways it's bizarrely wrong and in some ways it seems like a mystical crystal ball.

One thing I should've expected in those reports but didn't is that some maintainers end up nitpicking the reports instead of just doing the best to address the issues. When discussing the LLM code reviews, one author was arguing the security reports are really bad - it turned out that I completely disagree on the judgement they made. Instead of accepting that something can be improved, they argued how unlikely something is, how the project doesn't make this kind of guarantee, how that's an API user error... instead of saying:

- Yeah, we could check the parameter here. Currently we don't return any result so let's add a success/failure response rather than silently failing.

- This is a highly unlikely situation, but let's harden it anyway in case things change and it bites us in the future.

I don't know how many end up in that situation, but it was a whole category of people claiming the LLM reviews are bad, that I haven't really considered before.

Yeah, I don't think we can ignore the small issues anymore.

No matter how unlikely the issues seem to us, an LLM will happily chain dozens of them and do complex timing attacks to eventually gain access.

The post is making a statistical statement, not a universal one, and the examples it gives are of established codebases. If most reviewed code is older and well-used, I think we're seeing what we would expect to see (assuming the models almost always find something to report).