"As a white guy, I'd certainly encourage you not to worry about this -- if we're talking about an upper middle class, professional-ish activity, you can be fairly sure that all those white people feel self-conscious about their group being so white and would welcome you to join them."

This isn't an attack and your sentiment seems genuine but obviously this person knows his fears are not logical. I doubt most people can will themselves into feeling accepted even if others, like yourself, do your best to them feel accepted.

That's why representation matters