If you think that Klan members were ostracized from society (rather than being local centers of power) and that all they did was harmlessly rally outside of public places, you seem not to know the first thing about the Klan.

Even if that was just a hastily/poorly chosen analogy, there's still a point to be made: groups of powerful, immoral people do not stay on the sidelines of society for long.