Flagging amounts to anonymous moderation, which is why you see a lot of conspiratorial responses to it. Users have no way to know who is removing a post from acceptable discussion via flagging.

One possible way to address this is to make visible a list of users who flagged a post. The arguments against this are obvious. But without such information, in the end you have to accept one result of anonymous moderation is the generation of conspiracy theories.

It's a tradeoff, of course.