Yeah like when we bundled in a .js library for client side date processing that has a CVE affecting node.js servers with high score. Our auditors don’t care they tag the whole app as high risk. It doesn’t even run on the server!

the auditors that sign off on your security to meet your clients requirements usually know way less about your security posture than your clients do

its all just surface-level box-checking. most companies required to get 'penetration tests' just get an overpriced Nessus scan sold as a pentest and that meets their reqs.

while this is true it in no way diminishes the value that orgs like cve provide

Incompetent auditors don't detract from the classification system, though. If we removed every data point auditors misinterpret or don't care to understand, we may as well remove all metrics.

Most tracking tools have exception processes. But yeah, security as a product family instead of a simple score seems to be a foreign concept at most companies.

I feel that. So tired of management being completely uninterested in actual, actionable security holes but getting wildly spun up because they saw a notice with a big scary number that has absolutely no relevance in our architecture.

Don’t let the perfect be the enemy of good. It is(was?) a very useful and important system.

Trump must be receiving a lot of emails from companies wanting to fill the void, and I bet the Trumpiest of them all is going to be awarded a contract worth 10x the budget CVE had, and do a much worse job.

It’s Way Better than what we had before: software vendors making even arbitrarier decisions about how to classify them.

There are far too many bad actors for us to operate as an industry with no yardstick.

I disagree that it is Way Better than before. A judgement call is worth more than a team wasting effort chasing irrelevant pseudo-vulnerabilities being reported as vulnerabilities. A broken yardstick is worse than no yardstick.

But that's an issue organizations bring upon themselves, by defining semi-arbitrary KPIs that are used without proper interpretation. It's not directly caused by CVEs or assigned scores. It's like blaming git that it count lines in diffs, because your company created a KPI that measures developer's based on LOC changes.

Fair point. I was not blaming CVE for the situation, simply bemoaning the situation.

Spot on. Vulnerability scanners that make up an organizational Security Score (TM) tend to operate at the wrong level of abstraction, flagging some library somewhere that never runs and has nothing to do with your production flow or architecture, or some test keys with zero security impact. Go explain that to management, because obviously the security tools are right and you are wrong. This sad state of affairs is unfortunately the best that the security industry has been able to deliver. Trying to wrangle complexity by adding more complexity is the craziest notion to me. Yes, no scoring scheme is perfect, but when the scheme introduces more noise, what have we gained (well, security vendors gain, but what have organizations gained).

This is my research field. Do you have any input you can think of at the top of your head?

That's very cool. You probably know more about it than I do, then, but my advice is to articulate the exact problem you try to solve.

I expect your field is probably teeming with AI proposals or offers on how to manage vulnerabilities, but that is doubtful the way, because again it is adding complexity, and no classifier is perfect, especially when scanners fail to understand scanned applications and their threat models or environment.

Stop selling external scanners, start simplifying code? This will never work, of course, because security vendors sell the promise of security to those willing to buy it, in the form of add-on products and capabilities.

Empower people to ignore scanner reports without so much red tape? That would never work either, because megacorp wants compliance and reduced liability.

Build secure systems as opposed to cataloging and scoring flaws? That would never work, because building secure systems is hard, nature tends to favor otherwise.

Charge people for adding complexity and credit them for removing complexity? Sadly, there is no way to do that, especially since products must ship and quality is hard to observe, since it is often invisible and only surfaces when things are broken.

Off the top of my head, would be nice to require proof of exploitation, by adding CTF-like capabilities to apps, such that only if the flag is captured do we consider the report real. This places more burden on scanners, in that it is no longer enough to report an outdated library. Requiring some proof of exploitability reduces noise and increases SNR, reducing false positives. Naturally, not all vulnerabilities have working exploits, and scanners can never fully simulate an adversary, so we may get more false negatives, but at least we would not have to waste so much time upgrading pointless modules and breaking applications to appease a false report. So the idea is "here is a dummy asset, show me how you leaked or compromised it". Adding the dummy asset should be cheap, but would force scanners to better simulate an attack.

At the very least, there ought to be a knob to decrease scanner sensitivity.

And it's not enough to explain it to management, you also need to explain it to your ISO auditors, your customers et cetera ad nauseam.

Solving this problem in a generalized way is really hard.

Maybe I have a dependency on Foo which has a critical vulnerability in a feature that I don't use. I suppress the warning and all is well. Then two weeks later someone on my team decides to use that feature, not knowing that there's a problem with it. Now we're fucked, and we'll never know because the vulnerability has been suppressed.