I think Anti-Semitism is fairly rare (though it does exist). But it would be hard being one of the five Jewish families in my entire high school. I was good friends with them, and Christmas was a hard time for them. It would be very challenging to be a Muslim in good parts of America (outside of major cities; where, if we're being honest, it still isn't easy being Muslim).
It doesn't always stem from malice (sometimes just ignorance) but it very much exists.
And yes there are also some such issues in "Arab ghetto" neighbourhoods in Berlin and Paris but at least here I find it much easier to avoid by simpley not living in those neighbourhoods (& of course neither all muslims nor all working class europeans are anti-semitic, but you don't need a large % of the population to cause trouble/discomfort).
As a Jewish family in Germany (currently living in Berlin) I wouldn't want to raise my children in a small eastern community where they will be the only Jewish kids in school (I get the impression in North America anti-semitism is a pretty fringe phenomenon, but it's still "a thing" in Central Europe).
And similarly I probably wouldn't have wanted to be the only Muslim family in your town.