Well, I don't agree with that. You should respect the judgment of people who think that you're a bad person if they seem to have generally good judgment in other areas. Basically, this is the jester's point in Twelfth Night: "I'm better because of my enemies and worse because of my friends."
Indeed, but that's the point, isn't it?