You nailed it. Because we live in a society whose rules we'd like for the most part to uphold, we impose norms on ourselves and on others. Fairness is among the most important of those, and it's the foundation of many others.

Check out the Ultimatum game (http://en.wikipedia.org/wiki/Ultimatum_game) for another example of the same desire to keep people's dealings fair.