Immigrants don't need "our" permission to join "our" society, they only need the permission of the government, as a matter of legal formality. And of course one could argue that "illegal immigrants" are a part of society, just not legally, because societies aren't only defined in the context of states.

Also free speech is a universal value, is it not? It should apply equally to everyone, everywhere, all the time, and in all contexts.