Claude is perfectly capable of fixing these vulnerabilities; if that's what maintainers want, they can just ask for it. If the concern is cost, Anthropic offers a Claude for OSS program.

The reason Anthropic isn't proactively sending patches is, I suspect, that those patches would not be welcomed, because trust in the models isn't currently at the point where maintainers are willing to merge them without human review, which takes approximately as much effort as writing the patches themselves.

Anthropic's oss-scanner includes patches. We've received a batch and I'm working on some of it.

A good amount of the patches is straight garbage. It's not helped by the fact that a good amount (20~30%, maybe more) of the issues is also junk themselves.

I'll hopefully be able to share numbers (or data) after we're done with the batch.

70-80% not being garbage sounds like a good thing :)

I suppose it depends on:

- How many patches you receive

- How easy it is/long it takes to distinguish the good from the garbage

You're assuming the other 70-80% is actually good. It's not, there's a whole spectrum between "junk" and "good". A whole bunch of this stuff is "technically correct, technically even a security issue, but also just plainly not of actual importance".

You get free meals, 70-80% of them tasty on different degrees, and the rest burnt. That's OK because free meals, right? Except the small letters say you have to eat 300 meals a day.

I won't claim that my experience is representative, but what I've seen is closer to 50% being false positives, and then maybe 20% of the remainder being bugs but not vulnerabilities (aka lower priority bugfixes). The suggested patches have a much worse hit rate though, to the extent that I accepted only a small fraction after reviewing.

That said, an automated system generating only 30% true positives is probably still ahead on ROI as long as we're talking about tens of findings and not hundreds. The amount of human time spent investigating things that turn out to be false positives is the main difference.

Yeah, I was being "optimistic" in my estimate. And of course with a batch, you start at the high severity end, so it gets worse as you progress through. (Well, I assume at least, even some of the "critical" was just plain "not even a security issue" in our case.)

> If the concern is cost, Anthropic offers a Claude for OSS program.

thanks! It turns out that I am apparently qualifying for it and I was unaware

lets see whether Anthropic agrees

> Anthropic needs to teach their systems to propose a fix too. Detecting a coding problem and proposing a patch is what we should expect from a company with supposedly the best coding LLM.

Of course it's trivial to have an LLM generate a fix, too. They probably do. It doesn't mean a project (1) wants to take an LLM-generated fix at all, or (2) can immediately commit an LLM-generated fix without review (which takes time).

Punchline: Don't use curl.

Maybe not. And that’s totally fine.

But right now they’re not even offering one. They’re just putting the burden on the project without even trying to help fix it.

They have all the resources in the world, and they won’t even try.

And that’s of course assuming it’s a real bug, as we’ve seen a ton of them don’t seem to be. The last time we went through this, curl reported not only that a bunch of them weren’t bugs, but some of the bugs were listed previously on their mailing list, and that’s probably where the LLM got them.

So not only is it finding small bugs. And nonbugs. It’s reporting previously found bugs.

Pilling work on maintainers. Over and over again. To help? No. To make big numbers to show off for their IPO documents.

?

https://news.ycombinator.com/item?id=50448558

Could also be that most of the findings are low signal noise. Even the batch that was sent to curl was one medium and 39 low severity. One can only assume that the vast numbers of findings that haven’t been sent are even lower severity.

I'd be careful to take an examplary projects like curl and assume that level of quality is the average or even close to it.

Compared to curl, even the Linux kernel is garbage, and the Linux kernel is probably higher quality than all of the enterprise software that runs the world.