Surely one should be able to give a negative vote to somebody they didn't like?

That would also solve the issue with polarizing candidates, since by definition you now have to care that those who would never vote for you, would at least not vote against you.