Also, let's maybe not call the result impressive until we actually see what's under the hood.
These are relatively simple statistics. Some are even trivial. I don't see why it requires any people at all.
It wouldn't even have to be done by OpenAI. Given a list of vulnerabilities, it wouldn't be too hard for third parties to collect these statistics.
Who would put together that list unless individuals at OpenAI does it? It's not like we have machines that can think kind of like humans and do a bunch of stuff automagically.
> These are relatively simple statistics.
According to Greg KH 50% of the vulns are flat out wrong, or the patches don't fix the problem, or the scenario is impossible in the real world, or just isn't a security issue.
I have to wonder how many of these 1000 paths are simply dead drivers with security vulnerabilities [1]. Impossible to exploit because nobody uses it, but easy enough to juice up the numbers of "found" vulnerabilities.
I said in another comment that I await the technical writeup of the so-called new techniques that they aren't sharing for "safety reasons" but if history is anything to go on, they will never publish a writeup. They set out to get their headline and that's where the work stops.
They're not here to help the community, they're here to get headlines, and, possibly, replace the community.
Want the build the stats? Data mine the patches, and infer it yourself, publish it if you want to. That's how some people find bugs that aren't disclosed as explicit security fixes (and not every security issue needs a disclosure and fancy name, especially if it is introduced,found and fixed in a dev cycle before the code is part of a release)
- What subsystems are affected? - What's the median age of the vulnerable code? Is more recent code stronger than old ones? - What is the type of vulnerabilities found? Memory corruptions, logic bugs, race conditions, ...
This is an impressive result, but we need more info to understand how well this would translate to another codebase, and what lessons can be drawn from this experiment. Not every company has ~million dollards in tokens to spend.