Agreed. It's a subconscious power test by members of a group. Once they decide that you are stronger than them then they will leave it alone - and respect you for sticking with your principles. If they don't then, yeah, the group is toxic.