Anthropic's oss-scanner includes patches. We've received a batch and I'm working on some of it.

A good amount of the patches is straight garbage. It's not helped by the fact that a good amount (20~30%, maybe more) of the issues is also junk themselves.

I'll hopefully be able to share numbers (or data) after we're done with the batch.

70-80% not being garbage sounds like a good thing :)

I suppose it depends on:

- How many patches you receive

- How easy it is/long it takes to distinguish the good from the garbage

You're assuming the other 70-80% is actually good. It's not, there's a whole spectrum between "junk" and "good". A whole bunch of this stuff is "technically correct, technically even a security issue, but also just plainly not of actual importance".

You get free meals, 70-80% of them tasty on different degrees, and the rest burnt. That's OK because free meals, right? Except the small letters say you have to eat 300 meals a day.

I won't claim that my experience is representative, but what I've seen is closer to 50% being false positives, and then maybe 20% of the remainder being bugs but not vulnerabilities (aka lower priority bugfixes). The suggested patches have a much worse hit rate though, to the extent that I accepted only a small fraction after reviewing.

That said, an automated system generating only 30% true positives is probably still ahead on ROI as long as we're talking about tens of findings and not hundreds. The amount of human time spent investigating things that turn out to be false positives is the main difference.

Yeah, I was being "optimistic" in my estimate. And of course with a batch, you start at the high severity end, so it gets worse as you progress through. (Well, I assume at least, even some of the "critical" was just plain "not even a security issue" in our case.)

> If the concern is cost, Anthropic offers a Claude for OSS program.

thanks! It turns out that I am apparently qualifying for it and I was unaware

lets see whether Anthropic agrees