You have to earn 500 (or maybe 501) karma before you can downvote, and you're not allowed to downvote replies to you. The idea is, a person has to become a member of the community before they can just come in and downvote whatever they don't like, and you can't downvote someone for disagreeing with your point.

As far as I know, those are the main hard rules of downvoting.