Just relaying my personal experience, but I don't drink either and have never felt excluded. I occasionally get asked why (most people don't even notice) and I tell them the truth "I feel like crap the next day, even with just one drink".