I've said something to that effect to people I'm with because their actions in caring overly much what people think directly inconvenience me.

So suggesting that they don't care so much about something that doesn't matter hopefully makes them think about whether caring what people think is worth the effort.