> This could be a failure at any point in the bitcoin software.

No, it can only be a failure in the code that determines whether a block is valid. This is a small amount of code, subject to pretty intense scrutiny. And it can only be a bug that makes clients produce conflicting answers, not one that makes clients crash outright. (The 0.7 client, apparently, caught an exception and handled it by treating the block as invalid. We now know that this was a mistake; it should be made to crash instead.)

During the fork, 0.7 clients detected that they weren't on the longest chain and received an automatic message saying that their client might be out of date. People are now talking about building fork detection into merchants' systems, to fall back to safe mode if a fork is detected.

The code that actually determines whether a block is valid is not a small amout of code at all though - that's the entire problem! Whether a block is valid or not depends on subtle behavioural details of huge third-party libraries. The current issue resulted from a quirk of BDB and the developers still haven't managed to fully figure out when exactly it does cause a block to be treated as invalid. We also already knew that block validation inherits some of OpenSSL's quirks which aren't compatible with other, more spec-compliant crypto libraries. It's entirely likely that other code outside of the Bitcoin client also affects which blocks are valid.

I can't find what you mean by block validation inheriting OpenSSL's quirks. Could you explain what you mean? A quick google search suggests that Bitcoin uses OpenSSL only by passing it messages and fixed-size ECDSA keys for validation.

The BDB problem is not mysterious like you suggest, and the whole thing could've been converted from a fork to a DoS by changing a 'return false' to an 'assert'.

I forget the exact problem with OpenSSL, but it accepts particular invalid representations of signatures that the applicable standards say it shouldn't accept, because it doesn't treat the sign bit as a sign bit. This isn't generally a problem because those numbers should never actually be negative, but it's an issue for Bitcoin because everyone needs to agree on which signatures are valid.

That is a beautiful thing. So if I construct a block such that 100% of the clients fail on it, I can then have my own patched client control the entire BTC betwork, right?

No. You would have 100% control of the subtree following that failing block, but it would be very quickly outpaced by another subtree that the rest of the network would be working on. You need to construct a block that close to 50% of clients fail on, so that the network's work is split between two subtrees and they grow at about the same rate.

How do the clients recover when they encounter a block they cannot process and crash? Do they skip it and move on?

Well, if they crashed on a certain block then the operators of the client would notice and presumably update to a newer version of the client. The problem in this case was more subtle: the old clients did not crash on the block, they simply rejected it as invalid and continued working on an alternate subtree.

If you did somehow create a block that all clients except yours crashed on, you still wouldn't gain control of the network because the difficulty of creating a new block is calibrated according to the total power of the network, and only re-calibrates every few blocks. So for a while, you would still be effectively competing against the former power of the network, giving the client developers time to fix the crash.

You would need to find a very localized, specific class of bug in a piece of software that has been highly scrutinized over many years by cryptography and security experts who know there are millions of dollars at stake, and this bug would need to affect parts of the P2P network but not others, keeping in mind everyone voluntarily chooses which software to run. I think calling it "relatively easy" is a bit of a stretch.

  You would need to find a very localized, specific 
  class of bug in a piece of software that has been 
  highly scrutinized over many years by cryptography 
  and security experts

Couldn't you monitor the version control system trunk for patches fixing bugs in the critical section of the code, work out the bug based on the fix, and exploit it before everyone had their systems patched?

  everyone voluntarily chooses which software to run.

In things like operating systems heterogeneous software is often seen as a defence against bugs - but in this instance, wasn't it the differences between the 0.7 and 0.8 clients that triggered the split in the blockchain?

Surely the most secure thing is to be on the most widely used client?

Clients emit a warning in the GUI (and IIRC disable the RPC interface, which would be the fail-safe for merchants) when this kind of issue is detected (namely, conflicting blockchains of significant length received from different peers). So the most secure thing would be to be connected to a high diversity of peer nodes, and to pause transaction processing and check the news when you get a warning.

When I say relatively easy, I mean compared to the prevailing wisdom that causing a fork requires controlling 51% of the total bitcoin mining capacity.

Also, it doesn't need to be a crypto failure. Just a divergence of the clients on how they handle the protocol. Possibly a discrepancy that would be very unlikely to occur by chance or in testing, and only manifests itself when a malicious user triggers it. This is the threat model bitcoin has to worry about now...

You claim it is hard to do. However it accidentally happened recently. So it can't be that hard.

After all, if all "security experts" are so good, why didn't they catch the recent fork?

They didn't catch this because it wasn't a bug in the client code itself, rather it was a bug in a third-party library, which is currently being migrated away from. Further (as far as I'm aware) it's only happened by chance, never deliberately by a malicious entity.

Well it happened, with a simple decision to switch from BerkeleyDB to LevelDB. Luckily no-one malicious discovered it first, and it got fuzz'ed out.

Yes, and this issue is well known to anyone who's actually looked into the technical details of Bitcoin. It's really scary from a technical perspective - I believe that the developers have accidentally introduced similar issues before, it's just that no-one mined a block that triggered them.