I think there's a point here. The point being much more people talk about many eyes than actually audit existing code. The fact is, there are quite a number of bugs that stayed in the code for a long time despite being not that deep. Maybe that says we don't have enough eyeballs, but most of people quoting that sentence quietly assume we do, equating number of participants in FOSS communities with number of people actually auditing the code. Unfortunately, the former is much bigger than the latter. We pretty much have the same situation as we have in science publications where a lot of results are not reproducible and a lot nobody even bothers to check for reproducibility, despite reproducibility being the basic tenet of open publication and peer review and being necessary to validate the result. Unless there's a reason to - e.g. an identifiable bug - not many bother to go back to existing code and check it for correctness.

The quote is hyperbolic for the "all", and should probably use the word "shallower", but otherwise I don't see anything wrong with it.

"much more people talk about many eyes than actually audit existing code" is true of free software, but is also true of just about everything. More people think "I feel comfortable crossing this bridge because plenty of engineers have looked at it" than there are engineers actually looking at bridges. I haven't read the quote in it's original context for several years, but I don't remember it conflating software users with software developers.

The difference is that engineers actually know how to build safe bridges. I feel (and am actually) safe living in a house designed by a single competent architect, but I am not secure using a TLS implementation written by a single competent programmer. The only way we have found to end up with reasonable secure code is to have it reviewed by a lot of competent people.

There are a dozen points here... The shallow bugs statement is true when there is a community that is large and with incredibly deep understanding; linux and posix is a great example, there is staggering depth of knowledge on those interfaces and behavior. Unfortunately there isn't wide and deep knowledge of tls architecture and implementation, there isn't even a lot of commonality across interfaces.

There is a social axiom that you and I don't know crypto and we should leave it to the experts, yet they need help too.

There is conventional c style and this function, while documented, does the opposite. I had to look at the code a couple times to see the bug, a lot of reviews could have missed it.

There is conventional c style and the whole failure chain cleanup mess, look at that code again, they've got their own data representation and free function that uses it to detect if memory is allocated, the free is private but the initialize is done inline. That stuff happens everywhere by many projects, I'm not saying its wrong but it leaves it open for easy bugs, one gaffed variable declaration potentially screws everything and you need to know the data structure even though you don't directly use it.

And I don't want to pile on gnutls, I think their intentions are good and this is a bug, but this looks fucking prime for unit testing...

There are a lot of variables to determine and measure a projects health, the whole community needs to step up in the quality department, there are lots of ways to contribute and despite the adage, more people should look at crypto.

> I mean shit, I don't need to be an aeronautical engineer or work in the FAA to say that several people investigating an airliner crash will be more effective than one single dude sifting through a literal field of debris.

With zero people looking at airliner failures, nothing will be discovered, no matter how many inspectors the FAA has on its payroll. In the software world, there are almost no people doing code audits. Bugs and security holes are left in live code for years. Basically, Theo is saying "Shut up and audit."

How much open source code have you audited for bugs recently? How many subtle correctness issues have you found in projects you've looked at? For the sake of code quality, I certainly hope it's higher than the amount I've audited.

"Linus's Law" itself says little about whether or not people will actually look at code. It just says that more people looking is better than fewer people looking.

How much code I have audited, you have audited, ESR has audited, or hell, how much code Paul McCartney has audited, really has little to do with the obvious correctness and banality of the 'law'.

In other words, your interpretation of "Linus's Law" has no impact on code quality in the real world. Edit: However, since it's meant to imply that in open source, there are people looking at code, I think that the criticisms stand.

Yes, in the sense that Crito's Law (coined by me, now) "The bigger the ship, the more freight you can fit in it." has no impact on the real-world ship transport industry.

It is obviously true to the point of being banal. It is a pointless statement of uncontroversial fact that provides next to no utility to anybody. It's not even interesting for being a tautology.

If I were a particularly objectionable and self-promoting person, then perhaps people might object to Crito's Law whenever it were quoted on shipping forums, but that wouldn't make it incorrect. Nor would my shameless self-promotion make it profound.

(Is Crito's Law _precisely_ true? Well no, some large ships are not designed for freight after all... but the general principle is true.)

You seem to continuously miss the point he makes.

"Many eyes make bugs shallower" is indeed true and a tautology.

The way ESR meant it, it's merely BS.

He meant is as in: "because open source code is available for everybody to see, many people look at it, and so bugs are found more easily".

In the context it was said, it was meant as a factual observation about what GOES ON in OSS, not merely as a trite theoritical description about many eyes being better.

So, people are arguing against that, not against the idea that if more people ACTUALLY look, they will find more bugs.

The case is, very few people look at code. In some cases, even for extremely widely used software by millions of OSS users, even less people than the people paid to look at a particular proprietary software look at the code.

Heck, Gtk, the basis of most Linux desktops, had in the latest years like 1-developer really working at it (I know, because he complained publicly about the situation).

I don't know what happens in the Windows GUI toolkit or Cocoa, but I doubt there's one person doing all the work -- and only during his free time at that...

It is neither pointless nor content-less. It gets to the exact heart of the problems with ESR's claim. If Joe Shmoe had said it, people would not say "no shit", they would still say "no that is complete nonsense". Because it is in fact complete nonsense, for exactly the reasons Theo pointed out. All the eyes in the world don't amount to anything if they aren't being used to audit code. And to do that, they need to be attached to people capable of auditing code. ESR is simply being used as an example of his hypothesis being wrong since he is the most obvious example. It isn't attacking him to point out the flaw in his hypothesis.

"Eyes" obviously refers to "eyes of people auditing the code", not "eyes in skulls of people who are currently in the building", not "eyes sitting in jars in the Mutter Museum", not "every eyeball belonging to children currently living in Idaho". Furthermore, different eyeballs of course have different worth. Nobody in the world disputes that.

The law itself doesn't say anything about whether or not people will choose to examine the code. It just says that more people examining the code is better than fewer people examining the code. One audit is better than none. Two is better than one. Three is better than two. One shitty auditor is better than none. Two shitty auditors are better than one. One auditor twice as good as a shitty auditor is better than one shitty auditor, if you really want to get into tedious eyeball calculus. Etc.

Seems stupidly obvious; so obvious it isn't even worth stating? That's because it is.

Or to address your point another way:

If I said "You need fuel to make your car go." would you object that this is bullshit because not only do you need fuel, but it needs to be in your car, and the right sort of fuel? I don't think you would say, "But diesel fuel sitting in a puddle on the ground is useless and won't make your petrol car go anywhere!"

The point you seem to be deliberately missing is that the opportunity for something to happen and something happened are not equivalent. There is a natural assumption that because there is the opportunity for many people (however you decide to define "many") to review/audit/test code in open source projects, somebody must be doing so. Even when you base that assumption on the number of contributors to the project, non sequitur; feature addition/refinement does not imply bug fixing.

The NSA has lots of eyes, and they don't report the bugs they find. (Unless it's via parallel reconstruction...)

Another thought occurs to me. It isn't clear to me how much reading the source actually helps with finding most security vulnerabilities. It does seem like some of these recent ones in SSL have been found by detailed code review (after years of providing bypasses), but I mean in general.

It's remarkably hard to find a security problem reading the code unless that security problem is blatantly obvious. Part of the problem is that code also communicates intent, and security is in part counteracting that intent.

Most (more than 90%) of the security problems I found (almost all of which were in open source prjects) started off by observing misbehavior or wondering if I could induce misbehavior. This works because I would start off attacking an API, not the underlying implementation. Only after showing the problem would I look at the code. I think on one case, I was looking at code and thought "maybe that's a weak point." But in every other case, my testing was fully black-box to start (things later lead to a code audit in several cases).

> It's remarkably hard to find a security problem reading the code unless that security problem is blatantly obvious.

Then I guess I would consider most (C code) security problems rather obvious. The kind of stuff you see patches linked to in CVEs.. good old buffer overflows, off by ones, arithmetic screwups (mixing types or not checking for overflows), missing return value checks, the occasional swapped parameter, typo, simple logic whoopsie, etc.. These are incredibly common and rather easy to find once you get used to that sort of grunt work. But obviously you have to know what to look for.

The remarkably hard ones (for me anyway) are much more subtle, things like race conditions...

Using min where I mean max and max where I mean min is my favorite classic mistake that I usually just make most of the time and remember to check for and fix afterwards.

The same thing with misusing "this" in JavaScript closures: no matter how hard I try to consciously avoid doing it, it's always the first thing I check for when code doesn't work as intended, because I still make that same mistake all the time.

Another source of confusion in code that Ben Shneiderman pointed out, is that the meaning of "and" and "or" are opposite to programmers and normal human beings, so languages like SQL and logical expressions in most programming languages are fundamentally confusing to many people, and the inconsistency can be an un-obvious blind spot.

Normal humans mean to union when they say "this AND that AND those AND these", while programmers mean to intersect when they say "this AND that AND those AND these". So that's a terrible source of confusion.

In other words, programmers think adding "AND" clauses narrows down the selection like "((user.age >= 18) AND (user.gender = 'F'))", while humans think adding "AND" clauses augments the selection like "adults AND women".

I'm not advocating changing the meaning of "AND" and "OR", just pointing out that it's a source of confusion you should look out for, and be aware of when talking with non-programmers.

I saw Alvy Ray Smith give a talk about 3D graphics programming, and he confessed that he and his colleagues would just write some code and then flip the signs around until it did what they meant it to do. That made me feel a lot less embarrassed about doing that myself.

I try to read the code I need to use, but of course I never have the time to read as much as I should. It's really the best way to learn programming in general and to learn particular apis, as important as a musician listening to other musicians' music.

I feel much more confident using libraries whose source code I've already at least skimmed through. (Speed reading code and learning where to look for stuff later when you need it is a useful skill to develop.)

But reading static code isn't enough to trust it and be sure the comments and formatting aren't lying to you. Stepping through code in the debugger and looking at its runtime state and control flow is crucial to understanding what's really going on.

But the problem with reading code (especially code that you're not running in the debugger), is that you see what you think it's supposed to do, not what it's actually doing, especially when you're "skimming" over it as I like to do.

Occasionally I have the luxury of enough time to go into "study mode" and carefully read over code line by line (I've been reading the amazing npm packages in http://voxeljs.com recently, which is some amazing and beautiful JavaScript code that I recommend highly). But that is extremely tedious and exhausting, and uses so much energy and attention and blood sugar that I have to close my eyes and take little power naps to let my mind garbage collect.

And then I get these weird dreams where I'm thinking in terms of the new models and api's I've just learned, and sometimes wake up in a cold sweat screaming in the middle of the night. (I have sympathy for Theo and his neighbors he wakes up at night from nightmares about all the terrifying code he reads.) (So far no terrible nightmares about voxeljs, but a few claustrophobic underground minecraft flashbacks.)

Refactoring or rewriting or translating code to another language is a great way to force yourself to really understand some code. I've found some terrible bugs in my own code that way, that I totally overlooked before. And looking back, the reason the bugs were there was that I just saw what I intended to have written, instead of what I actually wrote.

And for those kinds of bugs, comments that describe the programmer's intent are actually very dangerous and misleading if they're not totally up to date and valid. Because the compiler does not check comments for errors!

I try to use lots of intermediate descriptive variable names (instead of complex nested expressions), lots of asserts and debug logs, and do things in small easy to understand and validate steps that you can single step through with the debugger. It's important to examine the runtime state of the program as well as the static source code. But that is hellishly hard to do with networking code in the kernel.

I also like to get away from the distractions of the keyboard and debugger, and slog my way through every line of the code, by printing it out on paper, going outside, sitting in the sun under a tree, and reading through every page one by one front to back, scribbling notes on the paper with a magic marker. That forces me to make my way all the way through the code before making any false assumptions, jumping around, and getting distracted. (ADHD Management Techniques 101!)

Don't let your mouth write a check your eyeballs can't keep.

And I'd rather be an asshole with eyeballs than a mouth full of bullshit.

Using analogies to avoid dealing with the actual topic is not productive. ESR claimed that open source projects are inherently more secure because "many eyes make all bugs shallow". That is what we are talking about. Not cars, not airliners. His hypothesis relies on the (false) assumption that many eyes are looking at the code simply because it is open source. That is not the case, as obvious security holes like these demonstrate time and again. It takes dedicated, qualified people spending significant time auditing code to make it more secure, not being open source. A closed source project with 1 security expert auditing the code is more secure than an open source project with 0 security experts auditing the code and a million users downloading the source and compiling it.