That's an excellent analogy. My wife doesn't drink, but we had some of my friends over for board games and some beers. By the end of the night she was quite annoyed and none of us could figure out why. She was annoyed that they were making jokes at my expense. She was mad that I was making jokes at their expense. She thought it was rude. Imagine being in a less trusted social situation than "wife and best friends", maybe a business meeting, where the client wasn't drinking but everyone else was. It's easy to see why there would be a social stigma against the one person who wasn't drinking, just like there would be a stigma against the one person who was if the shoe was on the other foot.