You are correct that such fears aren't terribly practical. Human beings in fact aren't terribly reasonable about what they are concerned about.

They bike to work, smoke a cig, and then worry about a Muslim terrorist blowing them up.

Unfortunately people aren't reasonable and our prosperity is partly based on resources and partly based on sucking in a lot of smart people from elsewhere to further enrich the nation we built with said resources and this shitty marketing could easily damage the second ultimately more important source of our prosperity.