Some are self-executing, by which I mean that they can be enforced "by construction" - if you create an object that doesn't follow the rule, other people will know. Said another way, breaking the rule would also require breaking some crypto.

Some rules, though, are social. For example, why does everyone try to extend the longest branch in the block chain? Sure, the protocol says it's the rule, but why should that mean anything? People don't follow rules because they want to. They follow rules because it's in their enlightened self interest to do so. If you could make money by choosing a different rule, somebody would do that instead. So it must be that these rules get followed because it's in the interest of Bitcoin players to follow them. The natural follow-up question is whether these social/economic rules are stable. That is, why not some other solution? Why not only extend blocks whose (nonce % 0x0d) == 0?

Cryptographers use a very particular notion of security in which they like the security of their schemes to "reduce" to a well-understood assumption. That is, we prefer it you can prove something like "if you can break my system, then you can also solve problem X" where problem X is well-known and widely thought to be very hard. Then either I am forced to believe that your system is secure or that you have found an efficient way to solve problem X. And since solving problem X is unlikely, I should consider your system secure.

As I said, some parts of Bitcoin do reduce in this way to known cryptographic primitives (which in turn reduce to problems we believe are hard). But not all the parts.