Because they radicalised each other and then went out into other subs no longer caring to hide their bigotry and racism.
If you you live in an apartment on a street with a nightclub on it, you're going to have drunk jerks on your street at night.
I can tell you as a former forum admin, that there is no way to avoid these people showing up. The rules don't matter, banning them doesn't matter, being a calm and relaxing place won't matter either.
What did matter is that the rules made sense and were followed indiscriminately by the mods and admins. It also helped to have a forum dedicated to spam and allowed for breaking of many rules (racism/harrassment and CP being excluded from the exception)
My banlist over a million users was 20 people long. I don't consider this sum to be hard to handle and at 1.2m registered users (without a need to register to read), I think maybe the problem is actually in the rules/application.
People hate feeling like they are being treated unfairly.
We would also enforce the rules when racism was targeting white people.
Well, many subs don't add a value to reddit. At least from my perspective and personal opinion. I don't understand why you cannot just unsub and don't read it.