The desire to prevent a Jewish person from being your neighbor is unethical and Bad. Such a "freedom" should not be encoded into law (as what happened in San Francisco with Chinese populations), or allowed to encode implicitly into societal norms.

People shouldn't be imprisoned or killed for being racist - but they should be scorned, shamed, and cause disgust when their viewpoints are aired.