News for hackers. Stuff that matters.
Re:The Great DOM Fuzz-Off of 2017 by drfuchs on Thursday September 21, 2017 @04:11PM (#15306924)
These results look pretty encouraging as to the general quality of the various browsers, until you read the fine print: "Only security bugs were counted in the results (doing anything else is tricky as some browser vendors fix non-security crashes while some don’t)." So, the numbers they show don't count random crashes, never mind incorrect behavior.

And who'd have thought that some vendors aren't even interested in fixing "non-security" crashes; I wonder which vendors those are? And I wonder how many "non-security crashes" their fuzzer found in each browser?

Even worse, they laud MS for the MemGC technology they use in their browsers, but MemGc is just a last-minute hack to try to keep use-after-free bugs from being security holes (by ignoring frees to anything that still has a pointer to it), without having to actually fix them. So, lots more bugs that really ought to crash instead go on to other undefined behavior, which hopefully doesn't amount to a security problem (never mind simply incorrect behavior).

So, it seems that their fuzzer is finding lots and lots more bugs, and they're just not telling us how many there actually were. Maybe the title should indicate better that they're limiting themselves to security vulnerabilities. I suppose there's a call for that sort of measurement, but it would be nice to also hear about the general quality of the software under test. The notion that they'd rate a browser that did exactly nothing as having a 100% perfect rating of zero bugs kind of rankles.

Re:The Great DOM Fuzz-Off of 2017 by svat on Thursday September 21, 2017 @06:10PM (#15307782)
Project Zero is exclusively concerned with security: when they say "bug" they mean "security bug"; when they say "fuzz" they mean "fuzz with an intent to find exploitable security vulnerabilities", etc.

IMO this restricted sense of "bug" is clear from the second sentence of the article ("in the recent years the popularity of those kinds of bugs in targeted attacks has somewhat fallen in favor of Flash (which allows for cross-browser exploits) and JavaScript engine bugs (which often result in very powerful exploitation primitives)"). Even the first sentence becomes questionable if we interpret "bugs" more broadly (Historically, DOM engines have been one of the largest sources of web browser bugs); the example given is a CVE, etc.

So IMO I wouldn't call it the fine print. And in this context, the remarks about MemGC also make sense (Microsoft considers these bugs strongly mitigated by MemGC and I agree with that assessment […] useful mitigation that results in a clear positive real-world impact) — "mitigation" is the term for something makes it hard to exploit a [security] bug, not something that makes the bug go away.

A similar question about general bugs (as in "the last bug in TeX…") would be very interesting, but I don't know if we have a way of measuring it.

Re:The Great DOM Fuzz-Off of 2017 by WhitneyLand on Thursday September 21, 2017 @09:48PM (#15308837)
Your’re missing his point. The issue is not the purpose of the effort, that’s clear, everyone agrees. The question I infer is closer to be something like: If crashing bugs are inadvertently found in software, from whatever source, what would you expect the interest/motivation/urgency to be in fixing them?

It’s a good question but I’m not sure the answer is clear. On small projects I’d make the case that 100% of reproducible crashes should be fixed, always. However when the “project” approaches operating system scale and complexity, most pure principle policies like that cease to exist for practical reasons. Most decisions, not just bug fixing, are about tradeoffs/cost benefit considerations due to the simple realization you can’t boil the ocean.

Re:The Great DOM Fuzz-Off of 2017 by microcolonel on Friday September 22, 2017 @02:41AM (#15309889)
> you can’t boil the ocean.

Call me crazy, but it seems like we have a responsibility to boil this ocean. If we don't, the quality will continue to spiral down into the abyss.

Re:The Great DOM Fuzz-Off of 2017 by WhitneyLand on Monday September 25, 2017 @04:59PM (#15334089)
Boil the ocean is an expression meaning do the impossible, so I don't think it matters much if we have a responsibility to do something we are incapable of. It does matter if we do the next best thing, and that's where the trade offs come in.
Re:The Great DOM Fuzz-Off of 2017 by pm215 on Friday September 22, 2017 @01:19AM (#15309568)
"And who'd have thought that some vendors aren't even interested in fixing "non-security" crashes"

That doesn't surprise me too much -- it comes down to prioritization, and often the crashes a fuzzer finds will be weird corner cases that don't happen in the 'real world' unless somebody is deliberately exploiting them. Since 'browser crashes safely' isn't a very interesting outcome for the deliberate attacker or a major issue for the user, it seems reasonable to me that a vendor might prefer to prioritize fixing other crasher bugs that telemetry says users are actually hitting, or fixing bad UI that annoys users, or any of the other thousands of issues in their issue tracker.

No software is bug free and no dev team has unlimited resource, so any decision to fix a bug is implicitly a decision not to fix some other bug. "We found this with a fuzzer" isn't an automatic ticket to the front of the queue.

Re:The Great DOM Fuzz-Off of 2017 by hannob on Thursday September 21, 2017 @04:18PM (#15306972)
> And who'd have thought that some vendors aren't even interested in fixing "non-security" crashes;

While I can't answer that I can tell you that I was very surprised Firefox showed little interest to fix this: https://files.hboeck.de/crashff.html (linux only, crash via notification api, bug open for a year)

Re:The Great DOM Fuzz-Off of 2017 by opaque_salmon on Thursday September 21, 2017 @04:38PM (#15307118)
It doesn't seem to crash my browser, but I'm assuming it's reproducible due to the existence of that page.
Re:The Great DOM Fuzz-Off of 2017 by ChrisSD on Thursday September 21, 2017 @05:00PM (#15307299)
I can reproduce it on Firefox 55 and 56 in Ubuntu Mate, it crashes after I allow the notification. It'd be useful to have a bug link so I can see what the status is. It could be an Ubuntu issue for all I know.
Re:The Great DOM Fuzz-Off of 2017 by ryacko on Thursday September 21, 2017 @10:13PM (#15308936)
My guess it is for specific versions of the Gnome desktop.
Re:The Great DOM Fuzz-Off of 2017 by mixedCase on Thursday September 21, 2017 @10:09PM (#15308915)
Running Firefox 58 (today's build) with a fork of notify-osd as notification daemon. Doesn't crash.
Re:The Great DOM Fuzz-Off of 2017 by pbhjpbhj on Thursday September 21, 2017 @04:41PM (#15307154)
That link is to crash FF? I thought it was to info about the bug.
Re:The Great DOM Fuzz-Off of 2017 by ChrisSD on Thursday September 21, 2017 @04:42PM (#15307156)
Do you have a link to the bugzilla report?
Re:The Great DOM Fuzz-Off of 2017 by errantblaze on Thursday September 21, 2017 @05:57PM (#15307696)