I always found it as the "how much my gut reaction matches with what the mathematical model suggests" kind of puzzle - useful for both exploring your own biases and the limitations of the model.

That's just what I mean, you end up examining your beliefs about what your gut reaction would be. Holding the knife in your hand and deciding what to do with it is when you find out your real gut reaction.

But I don't really know if we want to build a moral theory upon my gut reactions to things. My gut reactions are probably inconsistent. They are optimized by experience, culture, and evolution, not for things vastly outside the scope of common everyday situations.

I know I'm not a 100% selfless altruist. I don't donate all my money to charity and I'm not going to risk going to prison to sabotage a power plant. But that doesn't mean I can't aspire to being more altruistic or want the world to be more utilitarian.

Almost all of EY's posts are about how your gut reaction to things is often wrong, inconsistent, or leads to outcomes you probably don't desire. E.g. Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/) where people will pay more money to save 4,500 people out of 11,000, rather than 4,500 out of 250,000.

Right -- gut morality is strongly based on small, local interactions.

If someone invites you over for dinner and they have a live calf there, nursing at its mother, and they're intending to slaughter it in front of you, it's not the same (to gut morality) as if they slaughtered it the day before, and really not the same as if they just bought some veal at the supermarket (i.e.: paid someone out of sight to do it).

Likewise; buying something that was produced in a way that involved significant human misery and/or death feels awful to gut morality if the sweatshop, diamond mine, or whatever is next door. On the other side of the world? Eh... well this one is $6 cheaper...

Why should it be so different? Well, gut morality isn't logical, at all. It's worth a bit of effort to do better.