You should only shadow ban manually marked attackers. The one you are sure, very sure that are not legitimate users. This way you can't annoy real customers, as the shadow ban is not automatic and can't trigger on them.