Many years ago you did not need any karma to down-vote anything. That has changed a long time ago and for years I refrained to get higher karma just to regain the ability to down-vote.

It seems there is a 500 karma threshold for down-voting comments and another one to down-vote posts.

Edit: And another karma level to down-vote comments beaten to death.