Maybe because the lack of tolerance you're displaying is The Problem, and down voting is easier than dealing with it.

What if you're actually wrong and the other person is right? I'm sure it has happened before and will again.

How would you want to be treated if the roles were reversed?

Regardless; censoring, which is essentially what you're doing, can only push people further into their corners.

Tolerance of anti-semitism? How can that ever be right? I picked that specific hypothetical for a reason because I think the speech being “censored” here is inherently wrong, I.e it has no place in public discourse.

Tolerance of any opinion.

Much of what is now generally considered capital T true was once extremely controversial.

And censoring will always come back to chew your head off since you're hiding rather than solving the problem.