I said that if you don't associate with the crowds I'm talking about it doesn't matter.

Ironically pretending that we don't play social signaling games and critiquing others for playing them is itself a sort of social signaling game. These games can't be avoided because they are a fundamental part of human interaction, so much so that when we aren't good at playing those games for whatever reason we are unable to function properly in society and are pathologized as a result.