Sure, the moderators may end up looking one-sided, but it sends a clear signal as to what kind of behaviour is not tolerated. The other members can adjust to the new normal, and some will leave. Either way, you've cleaned up the site to some extent.