What I'm worried about more is people throwing around labels such as "xenophobe" and "homophobe" whenever someone has ideas that run contrary to their own.

Perhaps he has some ill-conceived notions floating around in his head. I doubt that justifies your final statement.