This probably isn't the full picture, but you can't vote down comments left directly in response to your own comments (for obvious reasons), nor can you vote down comments over a certain age or that are already of a very low score.