At that point the abusers who move in will realize you can't see them and will harass your members with impunity. That doesn't seem like good community management. Actually it seems like throwing new members into the deep end (exposing them to every bad actor you already knew about) and at some point I suspect the only people who join your site will be malicious.

You can't be harassed if you don't listen.

That is flat out not true. Someone constantly attacking your character to others is going to have an effect. Those other people may be influenced by what that person says, and that will affect how they perceive you, which will in turn affect your involvement in the project.

No, it is true. If a community doesn't listen to that sort of behavior, then it has no effect.

That is a mountain sized "if".

That's part of community building. A good community will reject noise and move on.