I understand this work was based on a dated snapshot. Will there be any continuous effort, scanning new PRs as they land, to prevent new weaknesses from being added?

It will be difficult to claim that the problem has been solved without an idea of how big the problem is.

Given that vulnerabilities are almost entirely unknown, you are inherently trying to measure the undetected.

Is there a way to hypothesize how many vulnerabilities there are and them make a prediction on how many of those will be found by a model to get a sense of how accurate the estimate is?

The errors bars will be huge, but it's better than "zero or more".

Also there are varying types, its worth getting it tested if you need to do a lot of work with/around it.

A system that admits THOUSANDS of such vulnerabilities is certainly flawed in several fundamental aspects

E.g., 948 results is a high number to review in a timely manner unless your team is very large or spending several months combing through the results

EDIT: nvm i shouldve kept reading, answered

> At this scale, manually investigating every report is impractical for our research team. We therefore focused on validating findings that a) were reachable by a low-privileged user, b) required no special capabilities, and c) produced measurable evidence of memory corruption.

The only exception I know of is Android. It would be interesting to know how many of these vulnerabilities work there. Can't be zero anyway...

Then the question becomes how many vulnerabilities Sashiko didn't catch (If they use/review Sashikos review)

(As someone who used to handle nonpublic Linux kernel security reports and is still very interested in this stuff, I’d love to review a small sample of these reports to get a feel for what kinds of issues are being unearthed.)

Also, please don’t stop at LPEs and information disclosures. If you find cases where Linux can be convinced to misbehave or where the implemented semantics don’t make sense, these are good, too. I’ve found plenty of interesting kernel bugs, including some juicy LPEs, and almost none were memory safety errors :)

And, well, since you didn't _specifically_ ask for questions about the blog post:

Why are you not quitting your job in protest and screaming at the top of your lungs about it, seeing so closely what these models are capable of that your company is creating? There exists now something that is able to find 948 kernel LPE vulnerabilities. How, exactly, will humans be able to control and contain, say, next year's models for a nonzero amount of time?

OpenAI has 5k employees now.

Yes, you see these headlines about ml researchers getting paid a ton, well thats because there's 5k of them on Earth.

Unless you're a VP or PhD you're not getting life changing compensation. And guess what, all those people can get similar compensation elsewhere without slaving away 4 days a week in an office in San Francisco.

Your average employee is not making an extra 750k. You know you can look at compensation online you cant just do vibes based analysis and just invent your own reality. Find me a position at OpenAI thats paying 750k compensation.

Would be nice if the page could be readable without.

...and then proceeds to never answer a single question. Classic.