Some organizations--and even though it's easy to mock Apple here, I do believe they're one of them--sufficiently appreciate that improving security minimizes risk, and so spend a decent amount of money on it. You could argue they failed here, but even Adam Langley who essentially "is SSL" at Google admits he's not sure they have a test that simulates an attack on a possible similar implementation error in their own code (though he does argue that such an error would have been caught in Google's code review process.)

In any case, an attention to security is certainly not something that is borne out of an organization's revenue or number of employees.

I can totally see this being "innocent 2AM mis-judgement call by a single employee", judging from what my peers tell me of Apple's corporate culture. I do think that Adam Langley's suggestion that code review would help is plausible, but it merely just means that more than one person has to make the same mistake in a judgement call. (It reduces the probability of such an error happening; it doesn't theoretically eliminate it.)

https://support.apple.com/library/APPLE/APPLECARE_ALLGEOS/HT...

We begin therefore where they are determined not to end, with the question whether any form of democratic self-government, anywhere, is consistent with the kind of massive, pervasive, surveillance into which the Unites Sta tes government has led not only us but the world.

This should not actually be a complicated inquiry.

http://snowdenandthefuture.info/events.html

Take, for example, the implementation Dual EC DRBG in the FIPS 140-2 certified OpenSSL module -- it was fatally flawed, and has never worked in practice. (It will be removed from the next version of the module in light of developments in the past year.)

https://lwn.net/Articles/578375/

Exactly. That sums up my feelings about it better than I could. Something so critical, something trivially easy to catch in a code review was not caught. And, the only scenario in which a code review might not have caught it: no code review. That's no code review for libssl.

If this is incompetence and not malice, it's incompetence of monumental proportions.

Code reviews only mean your code is going through two sieves instead of one.

Of course it helps. But there is no guarantee of anything unless the reviewer is incapable of making mistakes, in which case you could just ask him to write the code in the first place.

Did you look at the bug? I'll quote it here:

    if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)  
        goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)    
        goto fail;  
        goto fail;  
    if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)  
        goto fail;  

        if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)  
            goto fail;
        if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)    
            goto fail;
    -   if ((err = SSLHashSHA1.update(&hashCtx, &somethingElse)) != 0)
            goto fail;  
        if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)  
            goto fail;

    if ((err = ReadyHash(&SSLHashSHA1, &hashCtx, ctx)) != 0)
        goto fail;

    if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0)
        goto fail;

Diff35

  if (...) {
    goto fail;
  }
  if (...) {
    goto fail;
    goto fail;
  }
  if (...) {
    goto fail;
  }

  if (...) {
    goto fail;
  }
  if (...) {
    goto fail;
  }
    goto fail;
  if (...) {
    goto fail;
  }

I disagree. We are human. We are not divine. We do stupid things and we can write stupid infinite loop or off-by-1 and the bug will live for a decade. I am sure you have written a program which can be easily exploited and you know that it is such a trivial bug.

There are many many ways this should have been caught before it even left the building by numerous automated tools. Heck most modern IDEs will flag unreachable code in the editor so there is no real excuse for this. (Adding -Wunreachable-code to a sample project in Xcode immediately flags the next line).

The programmer is human which is why automated verification tools exist.

(AppCode's default inspections will also flag the issue providing you run them)

I am not familiar with running static analysis and looking output, only did very minimal undefined behavior santisizer detection.

Lax development is par for the course at these mega corps.

For example, according to the Wayback Machine, our sites page had an entry about it since 2009:

http://web.archive.org/web/20091123132855/http://www.chromiu...

http://www.reddit.com/r/programming/comments/mw2sn/googles_a...

You take the experienced people off the shop floor, and call them inspectors. So now the quality of product coming off the shop floor is worse.

Ann is the first inspector in the chain. When she's busy (and remember, she will be because quality has dropped) she might be tempted to let a few things go because Bob, the second inspector in the chain, is bound to catch them. That's what he's there for.

Bob is the second inspector in the chain. Sometimes Ann really churns the product through. Luckily during those busy times he knows she's already inspected the stuff, so he only needs to give a 10% inspection.

So you have worse product with more errors and leaky inspection.

Ideally you'd have a system with skilled workers and self inspection. That's okay for aerospace (which pays well) but not so great for lower cost product.

Not quite sure how this transfers to coding.

How about this:

owner/peer of the module has to sign off, and randomly select two more. One must be QA and one is another programmer who works on the module. We can also pick a "junior" level, but then that's probably not going to work since Apple employes programmers who have some years of experience already. Or we can pick someone who isn't directly working on that module, but have some qualification to do review. Mostly just asking "why are you having two goto, why return -100 here)

But I see counterargument: they will just listen to the author if he's senior or the owner who is also a senior. Their words carry weight.

This is why it's a big deal.

Why should we even bother talking about bugs like this anymore? Pure distraction.

You mean: had total control over all the original iPhones that they could get physical access to. (back when jailbreaking was extremely simple and common)

One of the sources: http://www.forbes.com/sites/erikkain/2013/12/30/the-nsa-repo...

Basically how it worked was they jailbroke your iPhone and installed spyware on it. Is it quite likely that today, 7 years later, they have a remote 0day to do the same? Absolutely. But there's no proof that "the NSA has total control over all iPhones".

Suppose a contractor was hired by the NSA to write an exploit with equivalent function. How could it be crafted more cleverly?