Downvoting people who are mistaken is different than downvoting people you disagree with. Something that is factually incorrect should be downvoted because it contributes negatively to the discussion.