How about an "outside community norms / not nice" vote on any comment? If a user accumulates too many of them (relative to positive votes?) their comments get pushed down.

I think it's different from the existing "flag", or at least how I understand it: Currently, I'd only hit flag if something is deeply offensive / illegal / spam. And I think flag only exists on posts?