I think it's really hard to draw that line, though. You "know" it when you see it, but others -- quite reasonably, sometimes -- know it and see it differently.

As an example, I'm fine downvoting an opinion that I find morally gross or anti-social, even though others (such as the person commenting) might think it's fine, and even agree with it.