I feel fine downvoting comments with high scores that I disagree with, especially if there is no particularly strong argument in them. If they have low score then I leave them alone.