Well, it's a fair point - if the only people who get banned are actual Nazis, everyone will get called a Nazi before too long.

Actual Nazis are rare. They were rare during WW2 and they're really rare now.