The author seems to have had a worldview in which bugs don't really matter if people aren't coming across them, and in which the difficult part of dealing with a bug is either reproducing it or getting from a symptom to a cause.
If you were in a world where those two things are true then I think he's probably right that "many eyeballs" would help a great deal.
8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.
He even attempted an axiomatic explanation:
Maybe it shouldn't have been such a surprise, at that. Sociologists years ago discovered that the averaged opinion of a mass of equally expert (or equally ignorant) observers is quite a bit more reliable a predictor than the opinion of a single randomly-chosen one of the observers. They called this the Delphi effect. It appears that what Linus has shown is that this applies even to debugging an operating system—that the Delphi effect can tame development complexity even at the complexity level of an OS kernel.
The longer paragraph doesn't seem to contradict the notion either. My impression (based on the "How Many Eyeballs Tame Complexity" chapter) is that Raymond thought that "debugging" means "fixing bugs".
If I were criticising this part of essay, I'd say the main weakness is that the things Raymond thought of as "taming complexity" weren't really addressing the hard problem of reducing the number of bugs.
If the objective is just to fix bugs that have been found, well, that doesn't really feed into this narrative. Also, ESR was making a very large number of claims about the efficacy of OSS, and limiting the scope to bugs already discovered would not really align with the rest of the goings-on at the time.
That's a qualifier for what follows it. Seems true enough to me. If there are enough developers to notice bugs, they will probably be found and fixed quickly.
Right, what about this one: the bugs are all found eventually, only not by the right person.
Just kidding :)
I was a developer at the time (still am), and if I'm remembering correctly, ESR was active in Slashdot and a few other places I hung out.
I took ESR's claim about bugs to imply that the quality of open source software would be greater than that of proprietary software because the number of people who had access to the code would inevitably result in less bugs. A lot of the discussions around C&B at the time were about software quality. I don't think anyone expected there to be zero bugs, just that there would be fewer.
I am not convinced it turned out that way, but that's an interesting discussion for another thread.
The key word here is characterized. That word is not equivalent to found.
Security vulnerabilities are unique in that they matter despite being unknown.
Other bugs are only important because of their direct impact on users. It's not unreasonable to take everything here and apply it to known bugs, and not to unknown vulnerabilities.
I think this thread overstates the reality of the very, very crude metaphor that illustrates 'why open source can work' - even if it is very flawed.
I think by now we all kind of know 'it depends' aka there's context to everything.
That would explain so much… https://gitlab.com/esr/gif2png/-/commit/a8a761561b2a071e7452...
I've seen cases where exploitation was complex, but even then, that tends to be because the full exploit is chaining together several bugs, each of which is individually simple and could be fixed by a drive-by contributor.
> In Linus's Law, I think, lies the core difference underlying the cathedral-builder and bazaar styles. In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out. Thus the long release intervals, and the inevitable disappointment when long-awaited releases are not perfect.
> In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena—or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door.
> It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out.
By this definition, bugs like Heartbleed were indeed deep: they were not randomly found by a million eyeballs in a few days, they were were found by months of careful scrutiny by experts.
The fact that a bug can be fixed with a one line change doesn't mean that it's a shallow bug. A one-line bug can induce a rare, hard to observe behavior in very corner circumstances. Even if users are hitting it, the bug reports can be entirely useless at face value: they can look like a crash that someone saw once with no idea what was special that one time (I bet you would find OpenSSL bug reports like this that were symptoms of Heartbleed going back much longer).
This is why you need a complex and time-consuming QA process to even identify the long tail of deep bugs, which no amount of casual eyeballs will replace.
> The fact that a bug can be fixed with a one line change doesn't mean that it's a shallow bug. A one-line bug can induce a rare, hard to observe behavior in very corner circumstances. Even if users are hitting it, the bug reports can be entirely useless at face value: they can look like a crash that someone saw once with no idea what was special that one time (I bet you would find OpenSSL bug reports like this that were symptoms of Heartbleed going back much longer).
> This is why you need a complex and time-consuming QA process to even identify the long tail of deep bugs, which no amount of casual eyeballs will replace.
This is the point in dispute. As far as I can see Heartbleed did not require any special knowledge of the codebase; a drive-by reviewer taking a look at that single file had just as much chance of finding the bug as a dedicated maintainer familiar with the specific codebase. The fact that it was discovered independently by two different teams, at least one of which was doing a general security audit rather than specifically targetting OpenSSL, supports that.
I don't think it's relevant that it could have been found by anyone. We know empirically that it just wasn't. It was found by security auditors, which is just about as far as you can be from a random eyeball.
Edit: An even more egregious example is of course the ~20 year old Shellshock family of bugs in bash.
Or that few people were looking.
> I don't think it's relevant that it could have been found by anyone. We know empirically that it just wasn't. It was found by security auditors, which is just about as far as you can be from a random eyeball.
It was found by security people with security skills. But those were not people closely associated with the OpenSSL project; in fact as far as I can see they weren't prior contributors or project members at all. That very much supports ESR's argument.
It doesn't. ESR's argument suggests OpenSSL should not hire security researchers to look for bugs, since all bugs are shallow and people will just quickly find them - the Bazaar approach.
What Heartbleed has shown is that the OpenSSL project would be much higher quality if it took a more Cathedral-like approach and actively look for security researchers to work on it, and make them a part of their release process. Because they didn't, they released with a critical security vulnerability for more than a year (and there are very possibly many others).
Especially in security, it's clear that some bugs are deep. Any project that cares about security actually has to follow a cathedral-like approach to looking for them. Releasing security critical code early is only making the problem worse, not better.
> ESR's argument suggests OpenSSL should not hire security researchers to look for bugs, since all bugs are shallow and people will just quickly find them - the Bazaar approach.
It suggests they should release early and often to allow outside researchers a chance to find bugs, rather than relying on internal contributors to find them. Which seems to have worked in this case.
> Especially in security, it's clear that some bugs are deep. Any project that cares about security actually has to follow a cathedral-like approach to looking for them.
No it isn't. I still haven't seen examples of bugs like that.
> Releasing security critical code early is only making the problem worse, not better.
How?
How is missing a critical security issue that endangers all encryption offered by OpenSSL for more than a year (giving attackers access to your private keys via a basic network request) "working in this case"?
> No it isn't. I still haven't seen examples of bugs like that.
If you don't think finding Heartbleed after a year in OpenSSL was the process working, how about finding Shellshock was hiding in Bash for more than 20 years? Was that still a shallow bug, or is bash a project that just doesn't get enough eyes on it?
> How?
By letting people expose their data for years with a false sense of security. By encouraging projects to think security is someone else's problem, and they don't need to really worry about it.
Rather than releasing support for TLS heartbeats that steal your private keys for a whole year, it would have obviously and indisputably been better if the feature had been delayed until a proper security audit had been performed.
That the bug was eventually found is in no way a merit of the model. People find security bugs in closed source software all the time. The NSA and the Chinese and who knows who else usually find them even earlier, and profit handsomely from it.
The fact that it was found by people outside the project is the system working.
> If you don't think finding Heartbleed after a year in OpenSSL was the process working, how about finding Shellshock was hiding in Bash for more than 20 years? Was that still a shallow bug, or is bash a project that just doesn't get enough eyes on it?
Yes it's a shallow bug. I mean look at it. And look at who found it.
> Rather than releasing support for TLS heartbeats that steal your private keys for a whole year, it would have obviously and indisputably been better if the feature had been delayed until a proper security audit had been performed.
How much auditing do you realistically think a project with a grand total of one (1) full-time contributor would've managed?
If the code hadn't been publicly released we'd still be waiting for the bug to be found today.
This happens all the time to Windows, to Intel's hardware architecture, even to remote services that people don't even have the binaries for. There is nothing special about people outside the team finding security bugs in your code. After all, that's also what attackers are.
> Yes it's a shallow bug. I mean look at it. And look at who found it.
If a bug that hid from almost every developer on the planet for 20 years (that's how popular bash is) is still shallow, then I have no idea how you define a non-shallow bug.
> How much auditing do you realistically think a project with a grand total of one (1) full-time contributor would've managed?
That's irrelevant to this discussion. Per the essay, even a company as large as Microsoft would be better off releasing anything they do immediately, instead of "wasting time" on in-house security audits.
> If the code hadn't been publicly released we'd still be waiting for the bug to be found today.
I'm not saying they shouldn't have released the code along with the binary, I'm saying they shouldn't have released anything. It would have been better for everyone if OpenSSL did not support heartbeats at all, for a few more years, rather than it supporting heartbeats that leak everyone's private keys if you just ask them nicely.
This is the point of the Cathedral model: you don't release software at all until you're reasonably sure it's secure. The Bazaar model is that you release sofwatre as soon as it even seems to work sometimes, and pass on the responsibility for finding that it doesn't work to "the community". And the essay has the audacity to claim that the second model would actually produce better quality.
That supports the point.
> If a bug that hid from almost every developer on the planet for 20 years (that's how popular bash is) is still shallow, then I have no idea how you define a non-shallow bug.
A bug where you think "yeah, no-one except the x core team could ever have found this". A bug where you can't even understand that it's a bug without being steeped in the project it's from.
> That's irrelevant to this discussion.
Disagree; that the Bazaar can attract more contributions is a big part of the point.
> This is the point of the Cathedral model: you don't release software at all until you're reasonably sure it's secure. The Bazaar model is that you release sofwatre as soon as it even seems to work sometimes, and pass on the responsibility for finding that it doesn't work to "the community".
Few people were thinking about security at all in those days, at least the way we think about it now; the essay isn't about security bugs, it's about bugs generally. The claim is that doing development in private and holding off releasing doesn't work, because the core team isn't much better at finding bugs than outsiders are. The extent to which a given project prioritises security versus features is an orthogonal question; there are plenty of Cathedral-style projects that release buggy code, and plenty of Bazaar-style projects that release low-bug code.
If anything, these are canonical examples of non-simple bugs.
For exploitation, or for patching, maybe. But characterizing these bugs is still clear and simple IMO. I mean, rowhammer is about as basic as a bug can get - the value of this bit in memory changed even though it was never written to.
You're letting yourself get dragged into some weird rhetorical places trying to defend this part of Raymond's essay.
Rowhammer was one of the examples the person I was replying to listed. But the same applies for spectre/meltdown/etc.. They're all issues that can be understood - perhaps not exploited, certainly not fixed, but understood that a bug is there - without any deep architectural knowledge.
> You're letting yourself get dragged into some weird rhetorical places trying to defend this part of Raymond's essay.
If anyone other than you talked like this they'd be quite rightly downvoted to oblivion.
I don't think it is clear how you would do this other than with hindsight. When the bug was discovered, if we avoid architectural knowledge of why the bug was occurring then what information would the discoverer have access to?
- Memory corruption is occuring during execution.
- We can't find any part of code that appears to corrupt the values.
Finding the bug requires finding the Rowhammer affect, which means looking isolating the instructions that are accessing similar uncached addresses, and then finding the pattern of accesses and cache flushes which triggers the corruption.
Could you explain how this would be simple, or do you have a different view of the bug complexity that you could explain?
> Finding the bug requires finding the Rowhammer affect, which means looking isolating the instructions that are accessing similar uncached addresses, and then finding the pattern of accesses and cache flushes which triggers the corruption.
AIUI you don't need to know the details of the caches though? I guess you could argue that knowing the physical memory layout is deep knowledge, but you don't need to be an expert on e.g. that specific chip or memory controller to find that effect, no? And fundamentally if you can produce memory corruption then you can know that's a bug without having to understand the details of the memory hierarchy or hardware.
So observing that Rowhammer causes buggy behavior may be simple. But finding the actual bug - i.e. this sequence of instructions on this data causes the memory corruption would be much harder.
Part of this comes down to how you interpret the "many eyes makes bugs shallow" but I suspect that no number of eyes would have found the Rowhammer bugs by looking at the code.
> We do.
which, just to clarify, i didn’t mean as “one of us contradicts the other” but that “we experience different slices of life which are guided by different priorities”. i very much wouldn’t want to be a security researcher in the open source areas i occupy now. likewise i wouldn’t want to use the development practices i experience in community open source were i instead building a fighter jet.
looking at the bizarre overlap between “open source contributors who like {Nix,Haskell/FP,Rust}” and “people i know who work for defense contractors” though, i’m actually optimistic that our distance could shrink in time.
> Even the high-level organizing notion of "cathedrals" and "bazaars", which should have been a lay-up, hasn't really proven out.
you and i live oceans apart. a year of working around NixOS and especially linux on mobile phones has me working in very bazaar-like systems — where every component is swappable/composable — and seeing/using Linus’ law daily — in that people from out of the blue will patch things i’d been unable to figure out and when i hit edge cases that look familiar enough in any software i install i will debug it & send fixes upstream.
i’m well aware the above is niche. on the other hand i’m fairly confident that niche would be inaccessible to most of us who are otherwise participating in it were it not leveraging these two concepts.